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ABS.TRACT 

. The plan described was developed to evaluate the , 
training and cost-effectiveness of the Automated lliatructional 
Management System (AIMS), a computer-based training and resource 
developflient system being developed by the Training and Doctrine 
CossBand (TRAOOC) for possible installation in its schools and , 
training centers^ The Punctional Description of the proposed AIMS was; 
examined to identify the hardware and software ^packages intended for 
developments Stuf flebeam^s CIPP Model (Context/ Input, Process, 
'Product) served as the theoretical J^ramework tor the^evs9iluation« 
Evaluation questions were formulated to guide the examination of the 
AIMS hardware/software, training management, courseware, training . * 
effectiveness, cost^ and implementati on/organ iasat ion factors. A \ 
Milestone Chart was prepared to indicate deadlines for the evaluation 
act4vit'ies7 and a description of* the staff required to implement the 
evaluation plan was prepared. Draft data collection instruments were 
devised with the caveat that revisions would be required to tailor 
them to the specific software packages operati'onal at each AIMS site;* 
A. reduced version this pltn has been prepared to evaluate the AIMS, 
test site at the V^SJ Army Field Artillery School, Ft. Sill, . 
Oklahoma. Six references are listed, and copies of the data ^ ^ 
collec\:ion instruments a^ attached. (Author/LLS) 
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FOREWORDf. 



i 

The Computer-Based Instructional Systems team of the U:S. Army 
^Research Institute for the Behavioral and Social Sciences (ARI) performs 
research and development in the areai of educalionaT technology with 
applicability to military education and training. Of special interest is 
research in the area of large sc^le computer-based in^tructicfnal systems 
The development and implementation of these systems is* seen ^ a solution 
to current Army training problems such as the management* of aelf-paced 
instruction*. ^ r 

This Research Report reviews the proposed functions of one such' 
system th^e Autoiaateid Instructional* Management System (AIMS) — 
and provides a formative evaluation plan to assess its management, 
training, and cost effectiveness*' The research effort is respo.isivs td' 
the requirements of RDT&E Project 2Q263744A795, "Hanpower and Educational ' 
Systems Technical Area of the FY 80 .\RI Work Program. • 




A-F0RMA3fIVE EVALUATION TLAN TOr THE AUTOMATED INSTRUCTIONAL MANAGEMENT 
SYSTEM (AIMS) ^ ' . • ' ^ 



BRIEF 



Requirement: ' 

^ To develop a plan to evaluate the training and' cost-effectiveness of the 
Automated Instructional Management System (AIMS) . The AIMS is a^ conjputer-based 
training and resource development syetem being developed by t'he Training and 
Doctrine* Command (TRADOC) for possible installation at its schools and training 

• centers 

t * •• ' • 

Procedure: * • ... * 

The Functional Description of the proposed AIMS was exatJLned to identify the 
hardware* and software packages intended for'dev€;lopment. Stufflebeaim's CIPP 
Model (Context, Input, Process, Product) served as the theoretica.1 framework for 
the evaluation* Evaluation Questions were formulated ^ to gul^ the. examination of 
the AIMS hardware/sof tware, training management, courseware; training effective- 

-ness, cost, ahd iiJ5)lemen tat ion/or ganizatjonal factors* A Milestone Chart w^.s 
prepared to indicate deadlines for the eyalimtiin apti^ties .fydts the' 
Qf data collection instruments through preparation the final report.; A 
delBcriptlon of the staff required to implement the .eyal^iatipn-lplan was prepared;. 

. Draft data collection instruments were deyised <fith the. guidance that revisions^ 
wptild be required to tailor the instrtiments tp. thfe. Specific s.oftware packages 
oi)erational at each AIMS site. ^vl< V/i .i^ / ' 



Utilization, of -Einaings4 --' ... v.-t • rV^r.rr-r^^? ' '^■'r;^:\rj^:i^li'^:-T ■ 

A reduced version of ..this plfto- haiB^bXeu' ^)^^^^ test 
site at. the US Arny /Fi^ia^flat:^Ue•^ shiibQXi';M'';\..^ '' 
evaluation at Ft, SiU: vitljj>ro?4 for "^f't§im;."M^ which; may be 

ia?>lemente4 prior tp the -^WS i^8t^latipnV:a^^^^ The 
present plan provides .for. •^ahalvaek nreV rihd nost^^^^ 
intention of providing 
.conq>ittcr^based 'instructional 
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A FORMATIVE EVALUATION PLAN FOR THE 
AUTOMATED INSTRUCTIONAL .MANAGEMENT SYSTEM (AIMS) 



BACKGROUND 



The Amny has made, a substantial commitment to self- 
' pace d instruction which en^jDles the student to progress at 
his/her own rate. However ^ if thfe student must wait -for 
equipment^ assignments y> tests f instructor aid^ and even 
other students r the benefits of progressing through a course 
at an individual pace are lost. For this reason^ cpmputers 
have had an increasing role in the modern training institu- 
tion. Computer-managed instructional systems provide the 
necessary data processing associated with self -paced Iti-* 
St ruction so that timefly information can be made available 
to^insjtructprs and students. 

\- 

In 1975^ the headquarters of the U.S. Army Training and 
Dbctrine Command (TRADOC) directed that self-paced training 
be implemented throughout its schools; At one TRADOC 
school,, the u;sT Army Field Artille^fy School (USAFAS) , ^'t . 
Sill, Oklahoma, the impact of' this mandate affected eight 
cotirses training 6 - 8,000 *stud^t3 annually. A complex and 
/burdensome manual processi for administering these courses 
emerged* Instructors who were trained to develop, review 
and deliver instrjaction, .were , required to spend most of ♦ 
their time performing clerical .tasks such as marking -check- 
sheets to indicate student progress. 

™ As self -paced -courses proliferated,, an urgent need was 

perceiyed at the USi^FAS'^and throughout TRADOC for a system- 
'^>ticX^thod tor memaging ajid. administering the new^self- 
paced training, in response*^ to this need, a feasibility 
study was conducted to *det^rmine the most appropriate com- 
pater-managed instjfuction (CMI) system needed to support 
BSJtPAS self-paced 1;rairiing. An analysis was made of the 
Versatile Training System (VTS)^ developed by the Naval 
^Weapons Center at China Lake, .California; the ^lemphis CMI 

^ System dcj*; sloped by the Navy; and the Computerized Training 
Syistem (013) developfed by the Array under the ABACUS project. 
The systems were rated '.on fivje criteria: 



Note , The authors wish to acknowledge the technical and 

support assistance of Ms% Judith Paris, International 
Public Policy Research Corporation. 

•> 
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. 11) Training support requirements which include man- 

agenient information, scheduling f distance, com- . 

. pSter-managed instrucJ:ion, -aids to author ing/de- 

velopers7 and modest CAI 'potential? 

' t, ■ . 

(2) -.Optimal hardware configuration? 

(3) Maintenance; ' ' 

(4) Continued personnel support; and . • " 

(5) Evolution, potential . 

The VTS received ta hi-^her average "Righted rating than the 
other ■ two sysrems because of its matc^i with the USAFAS 
priorities for training support requitements. (B.underson, 
1977). ' 

The VTS was developed by the Navy in 1972 under a 
competitively procured 'contract 'which provided total support 
"for" software, hardware, maintenance, and documentation. The 
VTS consists of a central processing unit (CPUj and associ- 
ated peripherals, optical readers, and terminals. The types 
and quantities of hardware and capabilities, of $of ty/are are 
determined for feach site's unique requirements. VTS-^o£t- 
ware supports the development of\training programs for 
specific ratings (MOS) , indicates resource configuration and, 
scheduling information, and produces a variety of reports. 
System support software enables up to 64 simultaneous users 
to interactively enter, j^e, and retrieve a large amount - 
of data.. ' ^ 

The Navy has made Vailable to TRADOC a VTS equipment 
configuration, software support for necessary modifications, 
a site manager, maintenance support, and contractor support. 
The 'TR2U30C version is termed the Automated Instructional 
Management System (AIIIS)/:. The AIMS is a training managpent 
system designed to provide automated data collection, pro- 
cessing, .and retrieval for personnel file^x^ diagnostic ^\ 
testing, .training Schedules, graduation pre<^iction, and a 
variety of course information. 

. '.The AIMS was installed at the USAFAS in b^ctober 1978 \as 
the. test 'Site for all TRADOC schools. The test; system ^ 
consists'', of a . Digital Equipment' Corporation {DE\) PDP-ll/7p 
computer'/ associated peripherals, additional masis storage in 
the form of a moving head disk, a magnetic tape drive, a 
high speed printer., terminals, and optical ^-eaderg^ AIMS 
applications software consists of modified VTS software and 
progiJams newly developed according to the USAFAS' require- 
ments. - 
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following software, packages are proposed for jraodi- 
ation and development on the AIMS: 



System Support 

# Personnel i . ,/ - 

• Test 'and Ej/aluation 
•^J,.ed{np^ter--Mana9.ed Instruction (CMI) 

' • Resource Configuration and Scheduling * >^ 
^ m Report 'Generation 

The Sygtem Support Software consists of thCvDEC^s^ 
Resource Sharing Timesharing System/Extended (IteTS/E) Isoft- 
ware.. The RSTS/E allows files to be created, updated,, and 
deleted from, the 'user's terminals* A Privacy Act Compliance 
software package has been developed in conjunction With the 
RSTS/E for the Ft. sill test site so that no unauthorized • 
software can be installed or modifications made to existing 
AIMS software ♦ 

^ Personnel Software is being modified/developed so that 
dat^ such as name, xank, SSN, MOS, training qualifications, ^ 
and past haessignments can be maintained for each student, 
staff, and faculty member • The personnel files WxXX be used' 
to create a personnel data base; maintain assignment and 
training histories; assist in-processing and oyt-processing 
~^nct±Dns; provide class registration; and ^prepaife; class 
rosters, completion certificates, diplomas/ and training 
reports • 

The Test and Evaluation Software is intended "for the 
creation and maintenance of test items and tests. The 
software will support the creation of a ina'ster t^st item 
bank, .generate tests from the igaster file, administer aiid 
score on-line tests, produce printed tests for off-line Use, 
score off-line tests by a mark-sense reader, and store 
^)rescriptions keyed to each test item to be delivered during 
on-line testing or by hard copy for off-line testing, 

Computer-Managed Instruction (CMI) Software is planned 
to manage single and multi-path progression through a selfr 
pac^d course. The progress of ^ach student will be recorded 
and reported to the instructor so that modifications' to a 
student's specific course components or path can be made. 
The software will also enable the prediction of module, 
lesson > and course completion times based on historical 
data. ^ 

Resource Configu ration and Schedulincr Softw are is being 
inodxf led/developed to control the availabilit^y of training 

* • ' • 



resources. The software will provide class schedules and 
enable the assignment of training materials (texts, test" 
equiianent, tools r training aids), on a time reserved basis. • 

I^eports Generation Software will be modified/developed 
to produce the reports and forms to interface with the other 
AIMS software packages. Instructional and administrative 
personnel. will be aided by the rosters, tests, grade sheets, • 
diplomas, and reports 'produced by the Reports Generation 
software. 

The previously mentioned hardware and preceeding soft- 
ware packages are planned for implementation as a two-phase 
effort* During Phase Ir selected soft\^e packages will Ije 
tested at the.U.S, Army Field Artille;ry School, Ft* Sill, 
Additional software options will be exercised during Phase 
II. The Management Information Systems Office (jMISO) of 
TRADOC*s Army Training Support Center (ATSC). requested that 
the U.S. Army Research Institute (ARI) i^pire a plan to 
evaluate the AIMS. This evaluation plan covers all software • 
opt4.ons proposed in the Systems .Consultants r Incr Proposal 
No.' •'73-78-024, Deverppment of an Automated Instructional Man- 
agement Systen for tt^e U.S. Army (Revised 13 November 1978) . 
The plan is responsive to TRADOC*s request and reflects the 
objective of the ARI) Work Unitr "Computer-Managed Instruction" 
^Project A795r Thrust 4, Task B, Worjc Unit- 2, FY80 Work Pro- 
gram) . ' 

The following sections of this Research Hepoxt describe 
the proposed evaluation, its conceptual framework, purpose, 
procedures, staffing, 'and reporting requirements.. 
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EVALUATION MOEfjlL 

The thecJjetical framewark gxiiding the proposed evalua- 
tion of AIMS is the CIPP- (CONTEXT, INPUT, PROCESS, PRODUCT) 
evaluatiori model (Stuf f lebeam, et al., 1971). 

The first component - CONTEXT - refers ^o the initial 
assessment of need and underlying rationale for the program. 
This assessment ^occurs prior to program initiation and 
results in statements of system deficiencies and needs. 
With regard to the AIMS, TRADOG's commitment to self-pacing 
underscored the requirement for an efficient training man- 
agement system. The need- at the USAFAS was documented by 
the ^procurement request for an automated system to overcome 
the problems in their manual procedures for managing self- 
paced training. Thus, the documerifeation for the AIMS pro- 
curement at the USAFAS should provide ali-ihe information 
necfessary to describe the Context which led^to the AIMS 
program, and within which its evaluationwTll itake place. • 

The second cc^ponent of the CIPP evaluation model - 
INPUT,- refers to the examination and comparison of alterna- 
txve system designs in meeting program needs. This activity 
occurs early iji a system^s conceptual stage and* iTiasults in 
the selection of one or -more approaches or designs to meet 
the needs and objectives determined by the Context evaluation 
The Input evaluation has been performed and documented in 
the study comparing the Computerized Training System, 
Memphis CMI, and the Vgjsatile Training System (Bunderson, 
1977). In addition^ Systems Consultants, Inc. (SCI) has 
documented the training management requirements of all 
TRADOC . service schools and Army Training Centers. 
I 

The third CIPP component - PROCESS - refers to an 
evaluation of how adequately a program's intended imolemen- 
tation plans arid procedures are beihg carried out. This 
information is used dur.ing a program's early stages to help* 
correct any deficiencies aijd iti^rove the implementation 
process and conduct of the program, with respect to' the 
AIMS, a Process evaluation should be continuously performed 
during the entire installation period at the USAFAS test 
site. • This should include moaitoring and documentation of 
implementation problems and their solutions . ^f formal 
assessment of the AIMS components by government acceptance " 
tests o£-hardware/software and querying AIMS users should 
also be included m the evaluation crtE the AIMS process. 

The fourth CIPP evaluation model component - PRODUCT - 
focuses on the outcomes of the program .and examine s them i n 
-relation to sat:isfying the needs identified "previously. The 
information gathered in this evaluation, as in Process 
evaluation, is Used for program modifications anti improve- 
ments. Process and Product evaluation interact arid should 



occur simultaneous ly» y?(hereas Product evaluation determines 
the extent -to which oi^ectives have been or are being 
attained. Process eyfeluation describes" the procedures that 
w^re actually implaniented , and this can provitJe an explana- 
tion for whether or- not the objectives were achieved. - 
Similarly r both types of evaluation provide feedback for 
changing a program's procedures. With respect to AIHS, the 
Product, evaluation should be performed once the hardware/ 
software system has become stabilized and operational. 
Measures of the AIMS** outcnmes^ should be obtained from on- 
line^records as well as instruments to be administered off- 
line. 

It is clear from .the above discussion of the applica- 
tion' of the,CIPP model to the AIMS evaluation, that data is 
avail2d>le and has been used for decision-making regarding 
the Context and Input of the AIMS. The next section will 
focus on the purpose of the AIMS evaluation regarding the 
collection of the needed Process and Product data. 
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EVALDATION PURPOSE/ISSUES 



. The purpose of 'this plan is to: Evaluate the ^effec- 
tiveness of\the AIMS in ineetinq'TIlrVDOC^ s training management 
needs u The appropriate focus for the Process and Product 
evaluation of the AIMS is formative* Operational and train- 
ing effectiveness data wxil, be. collected at.Jthe Ft, Sill 
test site so tfiat revisions can be made prior to AIMS dis- 
tribution ^urmy-wide» 

The goal of the AIMS i« -to provide overall management 
support for TRADOC training programs* AIMS is intended to 
provide managers, instructors, administrators^ and support 
personnel witk an interactive computer system with 'which 
they can obtain: 

♦ Student progress reports 
m' Training schedules ' 

m Predictions of student graduation Mates 

# ^ Prescriptions for^ remediation ' 

m Test administration . - \ - 

m Test scoring , ' ^ 

m Monitoring of student progress durxnqr' training i 

A -select i'on of AIMS software packages, identified -in. 
the previous section^ will be operatfioasl §t the USASAS. 
The two USAFAS courses which will be use^ tautest* -the AIMS 
are the Artillery Survey Specialist Course (MC3S--82C10} and 
the Field Artillery Target Acquisition Specialist Course ' 
{MOS 17C10K 

The approach taken in the proposed^ evaluation plan in- 
cludes t • following steps i ^ \ 

Ensure that all parties concerned 
know about, understand/ and agree 
to the propQsed evaluation. 

Validate the USAFAS training nian-^ 
agement needs to be met by the AIMS^ 

Examine match between needs and the 
AIMS* functions and characteristics 
using data collection instruments 
specific to the test site* 



Step 1. 

Step 2. 
Step 3* 



1 

Step 4^ Assess the AIMS operations in the 
, two USAPAS courses* 

Step 5. P^rbvida input for ^odificQtiona to 
' the AIMS at USAPAS. 

Step 6* I<eassess the AIMS operations in the 
'two USAPAS courses ^ ' \ ) 

Step 7, Provide input for modifications to 

the AIMS to. be installed at other ^ 
TRADOC schools* 

The evaluation will focus on the following components: 

I • Hardware/Software 

-II, Training- Mctnagement ^ . 

III. ^Courseware 

,IV. Training Effectiveness 



V. Cost 



VI. Implementation/Organizational Factors 

A list of 'evaluation questions has been formulated to 
guide the es{aInina€2^so of each- component. Theses-questions 
were suggested by Orlansky and String (1979) , S^iQel, et 
al., (iS78) and an- analysis of the AIMS' objectives derived 
from the USAPAS needs identified by Systeros Consultants, 
Inc, (SCI) These evaluation questions appear in t:he next 
section under their AI^iS component headings. 




. EVALUATION * QUESTIONS • 

Hardware/Software ' , 

^ I^l Is the AIMS hardware/software" tonf iguration ade-' 

quate for the Army*s training and resource manag^*- 
mojit needs at Ft. Sill (and>, other installations)? 

* 1,2 What hack-up capal^litdes are available during 
system dovm-time? ' 

II, Training Management 

II .,1 Does data exist on each system participant and is - 
access to this data limited to appropriate users? 

II. 2 Does a course catalog exist? 

•II. 3 Is a- list of training tasks maiijtained for e^^ch 
course? - ^ 

II. 4 Are tests administered, scored^ and recorded 
according to instructors' specifications? 

II. 5 ^ Are sufficient resources available to handle the 

student load? 
« 

'11.6 Does a catalog exist describing all training 
materials? 

* * » • • 

11*7 jSow are instructors, students, and materials 

assigned to courses? 

• • » 

II. 8 How useful are the AIMS-generated reports for 
instructors and training managers? 

How accurate is the graduation prediction function? 

11.10 Have in/out processing time and inconvenience -been 
reduced? 

'11*11 Has student registration time biei) reduced? 

11.12 Are all training reports (class rosters, etc.) pro- 
duced in a timely manr^r? 

IK 13 Are all required training records and paperwork 
produced by AIMS? 

11.14 Has the system been effective in monitoring stu- 
dents? 



-TTlv " Couf sreware ~* - i - - - j 

What procedure is used by the instructor to define 
the instfuctiona.1 environment and strategy appro- 
priate for each student? 

Til. 2. /Are tests constructed according to instructors' . 
; spec'xft^5B:tions? ' , . 

III* 3. How does the instructor determine the hierarchy of 
CMI lesson topics? 

III,^ What feedback is available for revising instruc- 
tional materials and tests? 

III»5 Are the instructors able to develop courses, • 

according to the Instructional Systems Development 
Model? 

' III. 6 How much time is required to prepare training and 

test materials , contained. in CMI courses? 

* « 

III .7 What special problems, if any, were encountered 
when inputting materials on-line? 



IV. Training Effectiveness . * 

IV* 1 What percentage of the students score above the *^ 
cutoff for 80% of the tests? 

IV. 2 What percentage-of-iiiie-studenfes fail to graduate" 
the CMI courses? 

IV* 3 What are student attitudes towards the AIMS 
courses? 

IV. 4 ^at are instructor attitudes '^toward the AIMS? 

IV. 5 What unanticipated by-products can be attributed 
to -the AIMS? ^ 

IV. 6 How well do CMI course graduates perform on the 
;job? 

IV. 7 What is the average length of time needed to com- 
plete the AIMS dMI pourses? 



V. Costs 

V.l What costs have been incurred to, develbp, deliver i 
staff, and maintain the AIMS? 

V 



ic ^'^ 



V.2 What are. the estimated costs to manua^lly admin- 
- ^ ister USAPAS self -paced courses? 



V.v8 What costs, have been avoided .by' use of the AIMS% 
at USAPAS? • ^ 1 



VI. Implementation/Organizationai Factors 

• VI^l What was -the status of ' the AIMS within USAFAS? 

^ .* ' ' 

VI. 2 Was there a^greement or understanding on- th^^ur- 
poses and goals of the AIMS prior to implemen-, 
tation? ' ' 

yi.3 Were there clear lines, of responsibility/author-\ 
ity within the AIMS program at USAFAS? 

VI. 4 Did instructors and other system users receive 
familiarization or other training cSn the AIMS 
as a "tool" for their benefit? 

VI. 5 Was the AIMS field test representative of cohdi- 
• tions that would exist un^er n6rmal operational 
conditions? 

* VI. 6 Were there frequent meetings of ' user and, AIMS pro- 
ject personnel to monitor the field test and en- 
hance mutual undej^statiding? 

VI. 7 Were key personnel assignments to the AIM^ CMI 
' courses sufficiently free of turbulence? 

• VI. 8 What^ special qualifications should the AIMS in-- 
structional support staff have? 



i 
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EVALUATION ACTIVITIES 



•V Evaluative data on the AIMS will be.collectted by two 
approachf^s. In • the .first apBafoach the software subsystems 
of the AiMS will be tested with data inputs Cduring' govern- 
meht-coridycted acceptance, tests and actual AIMS operations) • 
This will provide some answers to the evaluation questions 
concerning the hardwarfe/sof tware and training management of 
the- AIMS (Components 1 and 2).*^ .The seconfJ approach will be 
to obtain the USAFAS records and to ;qu6ry or ^interview, 
system users for information on courseware, training effec- 
tiveness/ costs / and implementation/organizational factors . 
•'(Components 3, 4, S^.and 6). This examination of records 
and interviews of system users will also provide' information 
for Components 1 and 2. * 

'The specific activities proposed in ^is evaluation 
plan are described belowv^- A schedule of these activities 
and associated milestones is s^hown in Figure 1. 



Activity l; Obtain Contract S.upport 



^ There is a need in any evaluation (formative or summa- 
tive) for all" concerned indivi^als^ to understand:- 

-Who and what is being evaluated 

0 How the assessment is to' be made * 

- • What criteria are to be used to judge the 
program \ 

• Who Is to do the dati collection, analyses, 
and interprdt^^tion 

• What decisions* are to be made as a result of 
the study. 

• How much time,' cost, and resources are re- 
quired to implement the evaluation 

Attaining this levfel of understanding by all concerned 
parties" requires a set^ of explicit activities that should be 
an integral part of the management of this evaluation 
effort* the purpose of this evaluation has been estab- 
li&hed as forqtative — the findings are to be used for the . 
improvement of subsequent AIMS implementations. ~ then this 
purpose must be clearly communicated to all concerned indi- 
viduals.. This Nisespecially true when the evaluators "are 
seen as '^"outsiders^S-j^In the proposed evaluation, ARI and 
its evaluation^ contractor could be perceived in this manner 
by the user (USAFAS) and/or the system developer (SCJ) . 
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• ACTiymEs ■' . 

1. OBTAIN CONTRACT SUPPORT 

2. DEVELOPMENT/MODIFICATION OF 
dATA COLLECTION INSTRUMENTS 

i DATA COLLECTION 

4. DATA REDUCTION/ANALYSIS 

5. INTERIM REPORT 

6. IMPLEMENPREVISIONS 

7. DATA COLLECTION 

.8. DATA REDUCTION/ANALYSIS. . 
9. FINAL REPORT , 



• MILESTONES: 

-INTERIM REPORT SUBMISSION 

J^^- FINAL REPORT SUBMISSION 



MONTHS AFTER RESOURCE COMNUTH^NT 



7. 8 



11 12 13 14 15 16 17 



A 



. FIGURE 1 

SCHEDULE OF EVALUATION ACTTVITIES/MILESTONES 
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Therefore /.it is proposed that a set of formal an.d 
informal meetings/briefings be arranged to establish working 
relationships between all participants* The purpose of 
these meetings should be two-fold: (1) to com municate the 
purpose, procedures, and details_of the evaluation plan, and 
(2) to involve the AIMS developer and users in setting 
evaluation priorities, contributing to data collection plans 
and activities, and determining how the evaluative informa- 
tion would be used tcimproVe the AIMS. Attendees at these 
meetings should represent all ini^erested or "concernel"* i 
parties to the. AIMS implementation. These include:* TRADOC 
(ATSC-MISO) ; USAFAS personnel (representatives from manage- 
ment, instructors, academic departments, departments of 
training development and evaluation, etc.); SCt project and 
site managers; and ARI and its evaluation contractor. 
Cooperation between all concerned is essential to the sud^ 
cess of AIMS and its planned evaluation.. One possible means 
for ensuring involvement of the system developer (Sci) in 
the formal evaluation is tx> cbntract for the development of 
software that permits on-line eva,Iaative data collection/ t 
analysis on the AIMS. " . ^ 

- • Once a degree of understanding and cooperation- has been 
. achieved,* and agreements are. obtained regarding evaluation 
'^priorities, the. specific evaluation procedures need to be 
^discussed. Arrangements should be made for scheduling and 
•supporting the data collection, analyzing the data, and 
rejporting the findings for use in modifying the AIMS. 

The outcome of this activity is the formal approval of 
the evaluation plan. This plan should, be, a formal document 
suR^lemerited by Meiricr^i^a of Understanding that detail in 
specific t^rms the results of the above-described meetings 
to obtain interagency cooperation and coor^inai^bn • in terms 
of contract support. 

« . « 

Activity 2. Development/Modification of Data Collection 
' ■ Instruments ', ; r 

Interview schedules and survey instruments will be 
developed and/or modified to collect data of interest 
Instrument development/modification will correspond with 
the AIMS evaluation components. Hardware/Software (Com- 
ponent 1) will be assessed by a_series of on-line tests of 
the AIMS subsystems. Prior to the government's acceptance 
of the.AIMS^ tests will be conducted to determine whether 
the system meets the contract's specifications. The data 
collected should indicate the operational status of the 
hardware and software. In addition to these tests, the AIMS 
subsystems will be examined in operation. The hardware/ 
software should not be tested until the AIMS ha? been in 
place and the staff is trained and experienced in its use. 
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The operational tests will also^ provide 4ata on Training 
Manag^ent (Component- 2) . System uSers should be polled for 
information on the operation* of the stibsystems . Survey • 
instrxxments from the Computerized Training System jCTS) 
Evaluation (Seidell et al*, 1978) have been modified to 
r collect, i-nformation on ithe Resource Configuration and :Sche^ 
duling Subsystem (Attachment 1) and the * Reports Generation 
Subsystem '(Attachment. 2) . 

Survey instruments are also available for Courseware 
(Component 3). Thfese instruments when modified can provide 
the means for collecting da^ta on the Computer-Managed In- 
struction {(Ml) Subsystem and the Test and Evaluation /"Sub-^ . 
system. The CMI Subsystem permits the creation and execu- 
tion of self-paced training. Two survey instruments and an 
interview schedule which have besn developed for*. the.CTS are 
appropriate ' for examining the development and revision of 
the AIMS course materials (Attachments 3^ 4, & 5). The crea- 
tion, itiaintensnce, and administration of a test item, bank 
are also portioijifs of ^ iihie. Courseware effort. Attachment 6 
provides initial guidance for the design of an instrmnent to 
collect data on the usefulness and quality of the AIMS- 
constructed tests. 

s The fourth component of the AIMS evaluation plan is 
Training Effectiveness. According to Seidel and Wagner 
' (1977), training effectiveness' can be measured* in terms of. 
the graduate's ability to perf6rm the tasks selected for 
training and the gradusfte^s ability to perform- on the** job. 
The former measurement is made at the school while the 
Idtter must be m^de in *the field. Orlansky and String 
(1979) , in an analysis of the training and cost effective- 
ness of military computer-based instruction projects, indi- 
cated that data on post-graduation performance has not been 
collected. -An. examination of the job performance taedback 
system at USAFAS is needed to determine what in^trtifnents and 
procedures have to be developed to make these measurements. 
In the evaluatiqn of the Advanced Instructional System 
(AIS) Dallman, et al., (1979) obtained supervisor ratings 
of AIS» graduates* job performance one -to three months* after 
graduation. Also, graduates rated the appropriateness and 
usefulness of their training. A similar assessment proce- 
dure could be adopted for the AIMS. 

Seidel and Wagner (1977) suggest the following 'types of 
training effectivehess measures? absence rates, accuracy 
scored, achieveinent measures, .attitude scales, attrition 
rates/ and trailing time. The AIMS is d^signisd to record 
within-course training effectiveness data. The CMI Subsystem 
provides a timekeeping function. Achievement and accuracy 
data are recorded by the Test and Evaluation Subsystem. It 
is anticipated that the Reports Generation Subsystem could 
produce data in thf* desired format. Attitude questionnaires 
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,will be needed for all types of AIMS users. Categories of 
system users include students, instructors, training spe- 
cialists, educational, specialists, instructional program- 
mers, battery clerks, registrars^ and the Commander, Stu- 
dent attitude questionnaires have been modified for the AIMS 
evaluation (Attachments 7 & 8)* ^ 

. It is positfible that all* data colleeticn instrvmients 
could be administered at the .AlMs terminals and tabulated by 
the Te.st amd Evaluatioh Subsystem. Some additional software 
development may be needed for>this capability. 

Cost effectiveness data (Component 5) will reqixire'the 
development of^ AIMS-specific data collection instruments* 
Some of 3bhe cost data Orlansky' and String (1979) recommend 
for collection include:, program design, delivery, manage- 
ment, per'sonnel, and eguipmentf'. The Cost-Effectiveness 
Specification for Computer-Based Trainilig S;^stems KSeidel 
and Wagner, 1977) can be *used^as a fejamework provided that 
^he appr6priate- cost' data are"^available and can be collected 
at^'USAFAS.- A list of cost categories derived from the / 
Seidel and Wagner Cl577P^ocument appears in Attachment 9/. 

• W^en performing an evaluation in an^ operational environ- 
ment, it is important to coit^le^iely document that'* environ- 
ment, attending to those fsrdtors, which probably infljiience 
results (pallman, et al-, A'979)/ It is clear that such 
- factors ras irtstrtictor bias , unstable system componefryts , 
unclear lines of responsibility, and vinrealistic student 
loads can negatively affect outcomes of computer-based 
instructional systems.- Structured interviews need /to be 
developed for use with the USAPAS management' and instruc- 
tional personnel, and with the, AIMS project personnel to , ob- 
tain information on implementation and organizational . fac- 
tors (Component 6). Work sample observations (similar to 
those employed by Dallman, et al., 1979, Attachment 10) can 
be used tc identj.fy the AIMS* 'instructional manpower require- 
ments. Irtplementatior and organizational factors influenc- 
ing the- success of a training management system can also be 
reyeaAed in instructor and staff attitude surveys (Attach- 
ments • 11 and 12) . 

A list of the data collection i^strunvents/procedures 
discussed in this section is presented*^ in Table / bn the 
following page. 

) 

Activity 3.^ Data. Collection ^ 

The questionnaires, student records, interviews, and 
other instruments described in the previous section will be 
initially administered at USAFAS during a four-*month period 
(as shown in Figure 1)-. This period is to coinci^ei0l7ith the 
implementation of ,the Artillery . Survey Specialist Course 

23 • 



^ TABLE 1 

EVALUATION COKPOHEWT/IHFORHATION SOURCES 



AIMS Evaluation Co{npon*nt 
I. Hatdwre/Software 



II. Training Management 
III. Courseware 



XV. Training ECiectiveness 



V. costs 



VI* ' Implementation/ 

Organisational Factors ' 



Information Sources* ^ 

* Adceptance Tests 

Systems Logs (Mainten- 
ance) 

Attachment 2 (Section IV} 
Attachment 3 (Questions 

8-10) 
Interviews 

Attachment 1 - Resource 
Configuration and ' 
Scheduling Survey 

AttachiT)ent 2 - Reports 
Generation Survey 

Attachment 3 - Course 
' Materials Developnent 

Survey 
Attachment 4 - Revised 

Training Materials 

Survey 
' Attachment 5, - Course 

Development Interview ^ 
Attachment 6 - Test and - 

.Evaluation Survey 
Interviews 
Observations 

Studen"* Records 

Achieveusnt scores and 
other measures as noted 
in Cost-Effectiveness 
Specitication, Vol. 3 
{np. 67-933 Seidei and 
Wagner, 1977.* , 

Atcacliment 7 - Student 
Attitude Questionnaire 

Attachment 8 - CMI 

Attitude Questionnaire 

Field Feedback Reports 

Intervi^^s 

Cost Data 
Time Logs 
Contractor Reports 
CosCr-Ef fectiveness Spe- ' 
cification (Vols. 1 & 
•111, pp. 5-66) Seidei 
and Wiigner/ 1977 
Attachment .9 - Coat 
-Categories 

Observations (Work . 

Sample) 
Observations (Attachment 
. 10 - Instructor Task . 

List) 

Attachment 11 - Instruc- 
tor Attitude Survey 

Attachment 12 - Staff and 
Faculty Attitude Survey 

Interviews 
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(MO? 82C10) ^and the Field Artillery Target Acquisition 
Specialist Course- (MOS 17C10) • The specific schedule for 
quest^lionnaire/interyiew administration will be estedDlished' 
soon (after formal approval of the evaluation plan. 

*As many instruments as possi'^le should be. administered 
and/or. tabulated on-line. It must be recognized that the 
school's goal is to train its students in a timely manner. 
Therefore, a back-up manual training management system will 
need to remain operational until the AIMS has been debugged 
sufficiently and Sbetbilized. 

4 Persons .engaged in the evaluation effort will need to 
work closely with USAFAS and SCI personnel during this 
period. Data ga^hB3f^ at thi3 time are to be used for 
immediate revisions or modifications of the AIMS. SCI 
and/or USAPAS personne:[. must be made awar,e of problems -as 
soon as they Occur. Such probleins and their solutions 
'Shoul'd -then be documented « 

Activity 4. ^ Data Reduction/Analysis/Reporting . 

Evaluation data gathered on-line regarding hardware/ 
software, training management, and courseware' (Components 1, 
2^ and 3) will need to be reported to those who« must use the 
"i^iformation to make changes, in the AIMS* Such^^ports 
should be produced automatically by the AIMS buV may need to 
be summarized foi presentat^ion*" ta the users. 

Activity 5. Interim Report 

This report' will documjent the formative evaluatiqn 
activities and findings that are recorded during the initial 
application of the AIMS to the two USAFAS courses. This 
report will also include: 

• Description of the initial AIMS operations 

m Comparison of USAFAS training manaqemeijt 
needs and AIMS functions 

• Discussion of lessons learned 

• List of recommendations for modifying the 
AIMS to facilitate its implementation in 
other USAFAS bourses 

Ac tivity 6. Implement Revisions 

Modifications and additions to the AIMS will be made 
base<f ort the suggestions of the Interim Report.- Research on 
maximxzlng the effectiveness of computer-managed training 
will be conducted at this time. * \ 
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Activity 1. Data Collection 

The same procedures described in Activity 13 above will 
be re-initiated for an Additional foujf-month peridd* The 
findings and recommendations described in the Interim Report 
will have been transmitted to the AIMS developers and users. 
It is expected that modifications to AIMS based on these 
recommendations will have been made, 30 the focus of. data 
cdlectioa ia this -secont? period will be on- the effective- 
ness of these revisions* It is possible that long-terra data 
such as job performance feedbacjk may be obtained during this 
period* Cost comparison data within the OSAFAS also be 
gathered. ' > . 

■ Activi.tv 8. Data Reduction/Aitalysis 

• This activity is essentially the same as. described in 
Activity #4 above. The focus will be on lessons learned- 
that can be us6d as guidance for the AIMS installations at 
other TRADOC schools. 

Activity 9. Final ^Report 

The formative evaluation will he completed with the 
submission and distribution of a final xeport that includes 
the following: ; • ^ 

• Description of AIMS field test at USAFAS . 

• Deacriptijpn of evaluation activities 

f * 

• Findings/conclusions for each AIMS evaluation 
component, 

* • Cost analysis and projections (if feasible) 

' ' ' 

• Guidance for subsequent AIMS installations at 
TRADOC. schools 

a , °" ^^^^ gathered and lessons learned during this 

Sield test, a dcjument will be prepared to guide the next 
generation of Army computer-based training systems in the 
management of - instruction. Whereas the. immediate results 

i? fu"^S^3f.^''^i"^'^.^°" ^^^"^ ^"-^ the AIMS implementa- 
tion at the USAFAS, the long-term use of the data is to pro- 
vide an empirical and, analytical basis for computer-based 
approaches to training delive^. ' 



COMPAR^TIVB ANALYSIS 



The present evaluation plan does not -include a compar- 
' ative study between computer-'-managed and aianually-managed 
'^instruction. The division of AIMS users into experimental 
and control groups is not ^'ecoiffinended .for its initial im- 
plementation at USAFAS fo'r three reasons: 

(1) The system is in a formative, relatively 
unstable condition and- thus, not represen- 
tative of othetr sites 

(2) As aims; is phased in, the manual mode is . 
simultaneously being phased out, and 

» " -J 

(3) The most value to be derived from an eval- 
^ uation at this ^tage is to iise the inform- 
ation to improve the, AIMS, 

Ho^aver/ some initial comparative data may be collected 
at the USfiPAS if- the tKo courses are still conducted in the 
manual mode* This data should be collected during the 
latter part of the proposed study period. By that time, 
- improvements will have made and more acc\H?ate cost and 
tifaining effectiveness estimates can be determined • These 
data should also be. collected at ther second and third AIMS 
installations so that a more complete cost and training 
effectiveness analysis can be made* The Cost-Effectiveness 
Specification for Computer-Based Training Systems previously 
discussed CSeidel and Wagner> 1977) could serve as the 
framework for these analyses • . . . 

The following section summarises the staffing required 

to implement the AIMS evaluation plan presented in this 

\ document. The qualifications of eacji needed staff meniber is 

^ indicated with a summary of job t'ask? and estimate of person 

imonrths required to accomplish the iob tasks* 




DESCRIPTION OF STAFF REQtllRED TO 
.-IMPLEMENT AIMS EVAIiURTlON. PLAN 



A - hlX activities up to and including Interim Report 
B - activities up to and ihcluding Final Report 



Estimated Person Months 



A 



B 



1 Senior R^earch Scientist - 
Evaluation Specialist 

Experienced in evaluation plan-* ^ 
ningr design r and methods^- par- ^ 
.ticulariy w^ith regard to computer- 
based' instriiction^ 
Tasks : AIMS evaluation planning 
ahd design, inter-agency coord«nr^ 
tion, xnamagement of evaluation ac- 
tivities, reporting. 

2 Research Scientists 

r 

Experienced in evaluation techni- 
ques, instrument development, in- 
terviewing , cost-effectiveness 
analysis, ^reporting* 
Tasks: AIMS evaluation instrument 
*velopment, data collection, con- 
:t structural interviews, data 
anfeilysis/in^ '^rpretation,. disporting/ 

I'Sijbe Manager - Research Scientist 

Experienced in Evaluation methodol- 
ogy, liaison -with military school 
personnel ♦ 

Tasks: pversee on-site data collec- 
tion activities, liaison between 
USAFAS, SCI, and ARI, reporting. 

1 Research Associate/Assistant 

Sxperie?>c^d in instrument develop- 
ment data collection, reduction, 
and analysis. 

Tasks : Revise, test, and admin- 
liter data collection instruments^ 
suramar-ize data^ support site man- ' 
ager. \ J' 
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Estimated Person Months 



A B - 

2 Programmers x ' 4 4 

Experienced in AIMS software siib-- ^ - • . 

system develppment* 
Tasks ; ^Prepare programs that pro- 
vide data collection instruments 
on-line ^ analyze data/ and tabu- 
.lute findings into properly for- 
•matted reports* ^ ii 

2 .Clerk Typists - • 3 4 

Experienced in typing data col- 
lection instruments / use of job 
entry data terminals. 
Tasks: Type^ reports / enter data • 
for analysxs^ type and reproduce ^ 
interim and final reports* * . 

This staffing plan assumes that all evaluatioi activi- 
ties are performe'd in a seventeen-month period; student and 
schbol records are made available upon request by USAFAS 
personnel; school and AIMS development personnel (USAFAS/ 
SCl. etc.) are available for on-site interviews as scheduled; 
and programs can be written to provide automated data col- 
lection, analysis, and reporting on the AIMS. 
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Attachment 1 



RESOURCE CONFIGURATION AND SCHEDULING SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



J- 
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RESOURCB-CONPIGORATION AND SCHEDPLING SURVEY 
AUTOMATED INS^UCTIONAL Mft^lAGEMENT SYSTEM (AJMS) 



DATE 



The/ purpose of this survey is to bollec^^ ^information 
on the allocation of resources within your ^area.^^ Th^ re- 
sults, will be consolidated with other surveys to determine 
the effectiveness of the Automated Instructional Manage- 
ment System (AIMS) • , ' * 

As you read the statements and questions ^ make a men- 
tal comparison of yqur training area prior to and after the 
implementation of the Automated Instructional Management 
System* * , ^ . 

iPlease place a check (V^) ma^k by your ppsition below* 



Position 



Instructor 
Section Chief 



Course -iChief 



Division Chief 



Education Specialist 



JTrainihg Specialist 
_Records Clerk 
Other " " • ' 
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• RESOURCE .CONFIGURATION AND SCHEDULING SURV^Xn 
AUTOMATED INSTRUCTIONAL MANAGfiMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOCt^ TO 
. INDICATE YCUR OPINION OF THE FOLLOWING 
S;f ATEMENTS. COMMENTS EXPLAINING YOUR 

SELECTION Will be appreciated. 


YES ' 


NO 


COMMENTS 

• 


1. The computer system has 
been able to accurately 
account- for the location 

each studerit within 
the course. 

\ 






i 

* • 

* 


2. The computer system has 
maintained an accurate 
accounting, of student 
posit^n vacancies. 






i 


3. -When the student completes 
a task, the system routes. 
" ^him/her to the 'next task'' 
without delay. 






o 

0 

• » 


4. Students are routed through 
' the course according to . 
' the predetermined (normal) 
flow.^ 






o 


5. Student routing has been 

accomplished with a mini- 

, . mum "of errors. 

• 
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RESOURCE CONFIGURATION AND SCHEDULING SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTE2L. (AIMS)' 



» 

PtEASrCHECKTHE APPROPRIATFBIIOCKTO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. . 


1 STRONGLY AGREE 


AGREE ^ . 


1 NEUTRAL 1 


1 DISAGREE 1 


. STRONGLY DISAGREE 


• 

COMMENTS 


6. When, student positions 
in the; next sequential 
task are filled, the 
student is routed to an 

-^^ alternate task r for 

which he/she has the neces- 
sary prerequxsites* 












• 


7. When the student completes 
a task and all student 
positions in the succeed- 
ing tasks are filled, the 
\ instructor is alerted. 






f 






f 


81 Routing students to the 
correSct student positions 
has jrequired. close moni- 
toring by the instructor?^ 
supervisor* 














9* The instructor is alerted 
when the ^student has com- 
pleted all the required 
tasks ^ 















. RESOURCE CONFIGURATION AND SCHEDULING SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT- SYSTEM (AIMS) 




10. Prior to'.tihe AIMS/ did accounting for student tima in 
.the coarse place an unnecessary burden "on the class- 
room instructor? ,Yes No 

Explain. 



11. Does the usefulness drived from using the computer for 
accounting for students* time outweigh the workload in . 

its, collection? Yes No 

Explain. 
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BESOORCE CONPIGORATIO^J AND SCHEDULING SURVEY 
AUTOMATED INSTRUCTIOljAL MANAGEMENT' SYSTEM (AIMS) 



12. What "is your fremk opiniori of the capability of the 
cooputer system to route and account for students 
within your course? (Please indicate- advantages and 
disadvantages) 



Advemtages 



Disadvantages 
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Attachment 2 



REPORTS GENERATION SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM 



(AIMS) 



1 
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REPORTS GENERATION SURVEY 
AUTOMATED .INSTRtJCTIONAL MANAGEMENT SYSTEM (AIMS) 



DATE 



The purpose of this survey is to collect information 
on the reports generated by the AIMS to h§l|) maintain stu- 
dent records, and provide information for managing self- 
paced training* This information wilX be analysed with 
the idea of making record keeping sim|)'ler and reporting 
more useful* 

Please indicate your position below by a check mark 
(VO • If more than one position applies, make additional 
check marks* 



Dept. Operations Chief ' C ourse Chief 



Dept. Education Special- ^Course NCOIC 

ist 

Dept* Course Materials C ourse Training 

Analyst r ' \ Specialist 

Pept* Records Clerk ' ^Course Section 

7 Chief 

Division Chief ^Course Instructor 



Division NCOIC 



The following is a glossary of terms which will assist 
you in answering the quesj 




• On-line Testing - Us[ing the computer to administer, . 
score, and record* 

40 Diagnostic Tests (quizzes) - An informal test. 

• Regularly - A^ regular times or intervals. 

# Occasionally - Now and then. 

• Seldom - On only a few occasions. 

# On-line - Interaction of operating a terminal with the 
* computer. 

# Downtime --An interval of time when the computer is not 
i^roductive. ^ I \^ 



REPORTS GENERATION SURVEY 
AtlTOMATED INSTRUCTIONAL. MANAGEMENT SYSTEM (AIMS) 



PLEASE CMECK THE APPflOPRIATE BLOCK JO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COiyiMENTS EXPLAINING^YOUR 
SELECTION Wia BE APPRECIATED. 

• 


1 STRONGLY AGREE 


1 AGREE ' 


\ NEUTRAL 1 


UJ 
lU 

§ 

< 


STROr^GLY DISAGREE 


* • 

COMMENTS 


SECTION 1. Weekly AIMS 
Reports . 

. The AI5!S has reduced l:he 
Jtime the classroom in- 
structor spends on stu- 
dent records « 














2. Weekly AIMS printouts 

enable rapid analysis of 
student accomplishment 
in relation to his/her 
^ peers • 














3. The ability to display 
-student records on the 
terminal 

a. provides effective 
feedback on student 
effectiveness 














b. saves time in 

assessment of. student 
nee^s. 

J 








• 






'4. The AIMS has had no im- 
. pact on student record 
keeping at the division/ 
department level. 










0 
1 


• 

t 
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REf»ORTS GENERATION SURVEY 
AUTOMATED INS,TRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



i 



PLEASE CHECK tHE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATEp. . 


Ui 

xu 
cc 
c 
< 
>• 

or 

O 
0) 


Ui 
cc 

< 


< 

cc 

.UI 


DISAGREE 


STRONGLY DISAGREE 


4 

COMMENTS 


5» The AIMS has 'been suc- 
cessful in routing stu- 
dents determined by their 
prior accomplishments in 
the course. 












■ c 


6« The AIMS reports enable 
course managers to 

a*" assess trends as 
they develop 










• 




b* update the instruc-- 
tional process with 
minimum delay* 


\ 












7. The ^ AIMS student record 
printouts enable the 
inst rife tor t6 

«■» 

, a» analyze student pro- 
gress 














b. provide individual 
assistance 












« 


p^rescribe remedial 
training • 
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REPORTS GENERATION SURVEY 

AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 

« 



PLEASE CH^CK THE APPROPRIATE BLO'CK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS* COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED, 


STRONGLY AGREE 


AGREE 1 


NEUTRAL 


i DISAGREE J 


STRONGLY DISAGREE 


COMMENTS 
/ 


8. Recording no-goes in the 
weekly report has assisted 
in managing *the student 
through the course* 






— ^ 










• 




> 




















% 


r . 










« 


• 
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REPORTS GENERATION SURVEY 

SECTION II • AIMS-Generated Reports 

For each type of report generated by AIMS 

(Title of Report) 
^ * 



1, 

2. 



Are you. ,f pillar with this report? 
(If No, turn page to next report). 

Is this report available to you? 



Yes 



a. 
b. 



Regularly available 
jOccasionally available 
"Seldom or never available 



3* How frequently do yon use this report? 



a* 
b. 
c. 



Regularly 
'Occasionally 
Seldom or never 



How accurate is . this report? 



a. 
b. 
c. 
d. 



No 



Generally accurate 
"Minor inaccuracies 
JMumerous errors 

"Cannot judge accuracy of report 



5» What is your opinion about the format of this xeport? 



a. 
b, 
c. 
d. 



Excellent format 
[Format is satisfactory 
"Format requires revision 

No opinion 



The information contained in this report is 



a* 
b. 



Essential 
"Non-essential 



The content of this report 



a* 
b, 
c. 



^Should not be changed 

JJeeds minor revisions in the content 

"Needs major revisions in the content 
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REPORTS • GENERATION SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 

^ , . ■ — . 

8» How useful do you find this report? 

y 

a. Serves intended purpose 

. Of marginal use 

c» Serves no useful purpose 

COMMENTS: 



9* The objectiye of the AIMS ^ ^Report 

is to provide the course ittcinager with timely infoi^a- 
ti^^n concerning student progress and achievement .r" ^ 

a* Hov; well do you think the AIMS ' 

Report has met this objective? 



Ha^ thf expanded AIMS R eport 

provided J-ou with a better management tool? 
Explain* 



Is the revised. AIMS R eport 

received in time to assist the course/division/ 
department in completing the student records? 
Explain* ^ 



d* Has AIMS improved the timeliness of the student 
graduation prediction? Explain. 



e* Should any additional items be included or deleted 

A from the AIMS ' ^ Report? If yes, 

explain* ^ ^ 



ERIC 



REPORTS GENERATION SURVEY 
^ AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



10 ♦ Does the AIMS provide your course (s) with the neces- 
sary operational reports? If not, please list and 

describe any additional reports you think are neces- 

*sary* . ' . 



11. What AIMS reE>orts are^.not necessary? List and ex- 
plain • 



SECTION III. Student Record, 

The student record file was created to provide real time 
access to individual student records, both on-line and 
printouts, as required by the primary instructor or course 
managers. ^ 

1. Does the student record contain the necessary infor- ^ 
mation to enable you to: « ' '-'^ 

a. Analyze a student's progress? ^Ixplain. 



h. Prescribe remedial training? Explain. 



c. Take action to. separate the student from the 
course? Explain. 



2. Have you experienced any problems in using the ter- ^ 
minal to call up a student's record? If yes, explain. 
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REPORTS GENERATION SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMEIW:, SYSTEM (AIMS) 



3»* When you call for a student's record oh-line^ is there 
any apprecieU^le time lag in obtaining the display? If 
yes^ explain. 



4* Has the printout of the "student's record been available 
on a timely baisis? If no^ explain* 



Does the student record contain the necessary data to 
support the counselling and guidance program in your 
co'ursfe? If no^ explain* " * < <• 



6* Dd you use the student record .priji^ut to support 
. faculiy board actions? -Plrease explctin* 

t 

?• What additional information do you ne^d in the. student 
record? 



SECTION IV* Computer Downtime an^ Updating "Reports. 

(Answer the following questions with 'a "yes"" or "no" and 

explain your answer.) ^ 

' * ' • * 

1. Does computer downtime iritemupt the normal flow of 
AIMS reports?' . • 
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REPOKJfS GENERATION SURVEY ' 
AUTOMATED INSTRUCTIONAl. MANAGEMENT SYSTEM (AIMS) 



2. Dic\ the computer downtime result in any of the follow- 
ing: V X " . 

a. Irretrievable, loss of student data? 



b. Delay, in stiadent graduation? 



c. Distortion of students' data on his/her progress? 



d. Hand processing of student data? 



9 



e. Overtime (extra) work for administrative personnel? 



f . Delay in training until the system was restarted? 



g. Explain any special problems not listed abpve. 



3. Did" the temporary delay of recurring reports adversely 
affect student training? 



\ 
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REPORTS . GENERATION SURVEY . 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEtl (AIMS) 



4* Do you consider computer downtime a major drawback to 



5^ Following computer downtime^ was it nepessary to laanu- 
ally recprS and input student- data into the system to 
update any of the AIMS-generatec? reports? If yes^ 
identify the report and explain the problems you had. 



6. Have you experienced any problems with these AIMS- 
^ generated reports because of 'loss of data during down- 
time? Yes ^ No If yes, identify the 

report and describe the problems you had. 

I 

y ■ : ■ . - 

?• Were any particular problems, not discussed above, 
encountered in up-dating reports following computer 
downtime? 



8. Was it necessary to maintain a dual set of manually 
maintained student records to insure continuity , during 
' downtime? 



9* PJLease list and explain any problems encountered with 
the AIMS-^generated reports not previously covered. 



4S 
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Attachment; 



COURSE MATERIALS nEVELOP^!ENT SURVEY 
AUTOMATED INSTRUCTIONMi -MANAGEMENT SYSTEM (AIMS) 



r 



ERIC 
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COURSE MATERIALS DEVELOPMENT SURVEY 
AUJJOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



DATE 



The pxitpose of this survey is to' gather information 
concerning the development of AIMS instructional materials* 
Please answer all items in this survey as f actualriy and 
completely as possible, and with complete candor. Your 
responses will be held in strictest confidence. If you 
feel 'that you are not in a position to answer a particular 
item because you have not been closely associated with the 
AIMS project, please cir61e the item number and leave it 
blank. Your cqmments or suggestions will be greatly 
appr ec la ted . 

Please indicate (v^) your pos .ion relative' to AIMS: 



instructor 

Instructional Programmer 



Section Chief 



Course Chief 



Division Chief 



Education Specialist 



Training Specialist 



Other 



r 
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COURSE MATERIALS DEVELOPMENT SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT .SYSTEM (AIMS) 



^>LEASE CHECK THE APPROPRIATE BLOCK TO 
INDJCATt YOUR OPINJOHOF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 


I STRONGLY AGREE 1 


AGREE 1 


1 NEUTRAL . ] 


1 DISAGREE 1 


1 STRONGLY DISAGREE 1 


• COMMENTS 


!• Instructional pArogram- 
mers must ^oversee the 
input of course materials 
by. en try specialists 
and/or Clerk-Typists* 


• 










* • 


2. Delays were experienced 
in entering instruction-- 
al materials on-line 
because of the shortage 
of Clerk-TypAsts. 












• 


3- Clerk-Typists experienced 
little or no difficulty 
in entering lesson ma- 
terial or logic coding 
into the system* 

- /7 








\ 






4* Logic coding of lesson 
materials has caused re- 
latively few problems 
when entering materials 
* into the system • 




i 






/' 


t 


5. Editing materials on- 
line has been compounded ^ 
by Clerk-Typist typing 
errors. 










1 
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COURSE MATERIALS DEVELOPMENT SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLGASE CHECK THE APPROPRIATE BLOCK TO 
NDICATETOUR-OPtNtON-OF-THrFOttOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 



7 



^ p 

6* Little difficulty was 

experienced when delet- 
ing or adding new or re- 
vised AIMS instructional 
materials on-line". 



COMMENTS 



A complete AIMS unit of 
instruction can be changed 
overnight elimii\ating 'x.ny 
delay in student progress. 



What problems were encountered when inputting arid "^sav- 
ing" lesson material during the process of entering 
courseware on-line? 



How much adverse effect, "if any, has computer down -time 
had on entering of materials on-line? Explain. 
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COURSE MATERIALS DEVELOP^ffiNT SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMKrr SYSTEM JlAIMS) 



ERIC 



10. Did waiting for sysftem restarts delay the preparation 
of instructional materials? Explain. 



11. What special problems did you encounter when logic 
coding lesson materials? 
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Attachment 4 



REVISED AND ALTERNATE TR/ ING MATEJIIALS SURVEY 
•AUTOtlATED INSTRUCTIONAL I-IANaGEMENT SYSTEM (AIMS) 



I 
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Attachment 4 



^VISED AND iALTERNATE TRAINING MATERIALS SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM {AIMSl 



it 



REVISED AND ALTERNATE TRAINING MATERIALS SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



DATE 



The purpose of this survey is to gather information 
cqncerning revision and validation of AIMS course materials 
as well as alternate training material requirements. 

Please answer all items in this survey with complete x 
candor. Your responses will be held in strictest cortfi- 
dence* The results of this survey will enable the revi- 
sion and validatipn of training materials to be more effec- 
tive* 

If y9u feel t^t you are nat in a position t;o answer 
a particular q^uesticn because you have not been closely 
associated with the AIMS project, please circle the i€em 
number and leave it blank. Your comments or suggestions 
will be greatly appreciated. t 

Please indicate {\/^) your position relative to AIMS: 



Instructor 



Instructional Programmer 
Section Chie'f 



Course Chief 



Division Chief 



Education Specialist 
Training Specialist 
Other 
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REVISED AND ALTERNATE TRAINING MATERIALS SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



< 

PLEASE CHECK THE APPROPRIATE BLOCK TO 
INDICATE YQUR OPINlCyjl OF THE FOLLOWING 
STA-fEMENT^OMMeNYs EXPLAINING YOUR 
SELECTION WtL BE APPRECIATED. 

■ V- 

^ : 


STRONGLY AGREE | 


AGREE 


NEUTRAL 


;| DISAGREE 1 


STRONGLY DISAGREE | 


COMr:!NTS 


SECTION 1. . Revision and Valid- 
ation of Training 
Materials. 

• 








\ 




• 


-% 

1. The AIMS provides more 
flexibility in revising 
instructional materials 
than does the self-paced 
system. « 














2. Little difficulty has 
revising 

a* Individual display 














b. Tests 














c. Units ot Instruction 














d. Flow of instruction 














3. The AIMS review process 
has reduced the time 
normally required for 
introducing new or re- 
vised materials into the 
classroom. ^ 
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REVISED AN^^ ALTERNATE TRAINING MATERIALS SURVEY •- 
AUTOMATED INSTRUCTIONAL MAl^AGEMfiNT SYSTEM (AIMS) 



— , 

PLEASc CHECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. , 

> 


LSTRONGLY AGREE ^ 


1 AGREE. I 


\ NEUTRAL 


1 DISAGREE 


STRONGLY DISAGREE 


t 

> 

'V 

COMMENTS 


4* The AIMS instructional 
materials can be intro- 
duced into the course 
without the usual print- 
ing requirement proce- 
dures* 








c 


1 






- 










' / 






A 




• 




Jo 














• 
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REVISED AND ALTERNATE TRAINING MATERIALS SURVEY 
AUTOMATED' INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) ' 



PJLEASE CHECK THE APPROPRIATE-BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPl/AINING YOUR 
SELECTION WfLL BE APPRECIATED. 


YES 


NO 


COMMENTS 


^ SECTION II Rack— iiD Tr;^rm*nrr 

* Materials 

5, Was it necessary to pre- 
^ 'pare back~t(|> training 

materials to ^e used ex- 
clusively to cover compu- 
ter doiSntime? -If yes. 
please explaiii. 




- 


> 


6. Were course personnel, 

other than instructional 

. programmers, ^uVed to pre^ 
pare back-up ^ training 
materials to cover com- 
puter down-time? If yes, 
please explain. 








'7. Did the requirement to 

develop back-up training 
materials' increase the 
workload for the course 
writers? ^S^^^j^s/^ ^hat 
percentage? 






( 


8. In determining cdurse re- 
source requirements, 
should additional per- 
sonnel be programmed to 
handle the preparation 
of back-up materials? 
Please explain. 








9. Do you think there is a 

legitimate* requirement . ^ 
to develop back-up in- 
structJ.onal materials for 
all AIMS instruction? 









REVISED AND. ALTERNATE TRAINING MATERIALS SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



SECTION III. General Suinrhary, 

10 . Have you encountered any special problems in preparing 
the AIMS instructional materials^ not noted when pre- 
* ^paring prior self-pace^ materials? 



11, How would you change tj^e present system of revising 
.lesson materials- to. insure ,^dequate back-up for the 
\ AIMS instruction? 



12. What problems "peculiar "to the AIMS were encountered 
when using small and large student .groups for vali- 
dation of training ijiaterials? 
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Attachment 5 



COURSE DEVELOPMENT INTERVIEW 
AUTOMATED "instructional MANAGEMENT SYSTEM (AIMS) 



The Course Development Interview provides a structure 
for the Interviewer gathering data from Instructional 

DevBlopersLi Each Developer selected for an interview wi^l 

be asked the' questions that appear on the following pages. 
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COURSE DEVELOPMENT INTERVIEW 
AUTOMATIiD INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



NAME: 



COURSE: 



a. Planning/ Course Outlines, 
Strategies 

b. Original Authoring' (writing^ 
typing./ coding) 

c. Converting Eixisting Materials 

4. Modifying Materials due to 
• POI Changes 

e* Reviewing/ Debugging, & Testing/ 
Materials 

f. Revising Materials 

g. Coordination 

h. Other ' ' . 



% of Time Spent 



'2 



a 



COURSE DEVELOPMENT SUMMARY SHEET 
AUTOMATED- INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



NAME: [ . ■ POSITION: 

DATE : 



Course (s) worked on: 

Course Section Position 

1. . ^ ^ 

2. ' 

3. ; 

4. " '__ _ ; " 

5. - ' 

6. 



-f 
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COURSE DEVELOPMENT INTERVIEW 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



NAME? DATE:. 



: What feedback is available for you to use in revising j^- 
- structional materials an^ tests? 



L_ — ^ 1 



How have you used it? 



What additional feedback would you like to have?_ 



How would you use it? 



ERIC 
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COURSE DEVELOPMENT INTERVIEW ' 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIM 

NAME: ^ COURSE: 




What special qualifications are required by the in 
structional support yj^taff? 

a* Instructional Programmers ^ 



b. Course Development Personnel 



c» Computer Service Personnel 



Entry Specialists 



e, • Other { specify) 
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COURSE DEVELOPMENT INTERVIEW 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



NAME: 



COURSE: 



What administrative and personnel costs were in- 
curred to establii^h inservice training .programs? 
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COURSE DEVELOPMENT IJNTERVIEW 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



liAME: 



COURSE; 



What difficulties have been encountered in fitting 
the previously developed self-paced instructional mater 
ials into the AIMS ' management structure? 



67 

68 



COURSE DEVELOP^!ENT INTERVIEW 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



NAME: 



COURSE! 



What special problerals, if any, were encountered when 
entering (inputting) materials on-line? 
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CCimSE, DEVELOPMENT INTERVI|:W • 
AUTO^^ATED INSTRUCTIONAL MiiNAGEMENT SYSTEM (AIMS) 



KAMK: COURSE: 



What was the number and percentage of students who 
failed to graduate from each AIMS course? 



£"1 
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• COURSE DEVELCPMilST INTEPA'IEW 
AUTOMATED INSTRUCTIONAL iMANAGEMENT SYSTEM (AIMS) 



NAME: 



COURSE : 



What is the average training time for each AIMS course? 
How many graduates were, there in each course? 
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Attachment 6 



TEST AND EVAIiUATION SURVEY 
iUT3|0MATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 
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, TEST AND EVALUATION SURVEY 
AUTOMATED ^STRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



. DATE: 

The purpos'e of this survey is to collect information, 
on the tests produced by the AlMs. ^ The results wills be ^ 
consolidated with other surveys to determine the effec- 
tiveness of the AIMS.* Your responses will be held in 
strict confidence. 

Please place a ^dheck (V^) mark by your position be- 
low. ^ 

Instructor 

Instructional Programmer 

Section Chief 



Course Chief 



Division Chief 
Education Specialist 
Training Specialist 
other 



/ 



7: 
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TEST A^D EVALUATION SURVEY 
AUTOMATED INSTRUCTION L MANAGEMENT SYSTEM (AIMS) 



/ 

PLEASE pHECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 



1. On-line pretests are 

effective in diagnosing 
, student knowledge of the 
instructional unit. 



2. The results of the pre- 
tests are good predictors 
of student accomplishment. 



On-line pretests are not 
a factor in routing stu- 
dents through the course. 

\ 



On-line pretest results 
have little impact on 
student perfonrance. 



til 
< 



Since pretests. are 
optional, most students 
elect not to take the 
pretest. 



COMMENTS 



ERIC 
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.V TEST AND EVALUATION SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



I 

please check the appropriate bloc,k to 
indicate your opinion of the followin& 
Statements, comments explaining your 
selection will be appreciated. 

i 
; 


STRONGLY AGREE | 


1 AGREE 1 


NEUTRAL 1 


DISAGREE ' J 


STRONGLY DISAGREE ) 


■i 

COM.VIENTS 


6. .Of the students who 

elect to take the pre- 
, tests, the majority 
^ fail. 




- 








i 


7. Posttests have been ' 

effective in measuring 
student accomplishment. 












• 


8. Dis tractor counts have 

identified weaknesses in 
the instructional mater- 
ials. 














9. Pretest and posttest 

distractor counts have, 
pinpointed deficiencies 
inherent in the test 
questions . 












r 


lOV Posttests have proved 
/ to be highly success- 
' ful in routing the stu- 
dent into his/her proper 
learning alternative. 















TEST AND EVALUATION SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING VOUR 
SELECTION WILL BE APPRECIATED. - 



11. Distractor counts have en- 
abled instructional prp- 
grammers to make timely 
' revisions to questions and 
V- training materials. 



12. Because of misspelling and 
improper phrasing, more 
unanticipated responses 
... have^ surf aced than* ex- 
'''pe^ert^. \ 



13. Unanticipated responses 
are useful when revising 
instructional material ^ 



COMMENTS 



ERIC 
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Attachment 7 



STUDENT ATTITUDE QUESTIONNAIRE^ 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 
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' STUDENT ATTITUDE QUESTIONNAIRE 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIWS) 



This is a questionnaire to gather information about 
the Automated Jnstructional Management System (AIMS) • ^ 
There are no ^ight or wrong answers. Rather, we are in- 
terested in your candid opinion of the following state- 
ments. Your complete ' frankness in recording, your opinions 
will be greatly, appreciated. Individual responses will be 
held ii: strictest confidence T 



O 
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STUDENT ATTITUDE QUESTIONNAIRE 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK IH^ APPROPRIATE BLCiCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 



SECTION I. 



ERIC 



Course Content and 
Instructional Media 



' The objectives of the 
course are clear and I 
,^ know what' is expected 
of me. 



2- .'The material in each unit 
is organized in a way 
that I can learn. 



The overall course con- 
tent holds my interest. 



4. The lesson material 
makes you think. 



5* I cannot learn what I want 
to leairn with this kind 
of instruction. 



COMMENTS 




STUDEN-r ATTITUDE QUESTIQNNAIRE 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS)/ 



L 

PLEASE CHECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 


1 STRO/JGLY AGREE | 


/ 

Ul 

cc 
o 
< 


NEUTRAL 1 


I'disagree I 


STRONGLY DISAGREE | 


COMMENTS 


• 

6. Generally, the lessons 
are hard to understand. 












if 


?• .Generally, the lessons 
' are too long . 

« 








) 

t 






t 

8. The level of reading 

skill required in most 
lessons is too high. 












t 


9. Performance examinations 
cover what is pra^sented 
in the lefesons. 

• 














10. Generally, 'the Mssons 
seem to be planned just 
for me. 

• 






« 






f 
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STUDENT ATTITUDE QUESTIONNAIRE 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIAtE BLOCK TO 
INDICATE YOUR OPI^!:CN OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 


1 STRONGLY AGREE | 


AGREE {| 


1 NEUTRAL II 


1 DISAGREE ] 


STRONGLY DISAGREE | 


. '^■"COMMENTS 


11, I learned the course 
material very quickly 
using this method of 
instruction. 








■ 




■ 


12. I find myself hurrying 
through a lesson to get 
it over with rather than 
trying to learn. 












- 


13. I answer questions wrong 
intentionally (pretest, 
posttest) in order to 
get mor6 instruction. 












r 

<s « 


14. I waiste no time using 
this method of instruc- 
tion. 














15. I do my best as a result 
of this method of in- 
struction. 
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STUDENT ATTITUDE QUESTIONNAIRE 
AUTOMATED INSTRUCTIONAL MANAdEMENT/SYSTEM VjjMS) 



r-.-- ' _ 

PLEASE ^HECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEiyiENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 


STRONGLY AGREE 


AGREE 1 


< 

>- 

Ul 

z 


! DISAGREE 1 


STRONGLY biSAGREE 


COMMENTS 

\ 


16. I Always know how well .1 . 
am doing in this course. 






CI* 








n. The AIMS^is a very effec- 
tive method of instruc- 
tion. 

<• 






* 








18. I feel that no one really 
cares whether I learn or 
not usilig this method of 
instruction. 




- 










.* / 
19. I ffeel that I am pushed 
' too rapidly through the 
lesson material. 

» 














20. An instructor is readily 
available for assistance. 










) 
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STUDSNT ATTITUDE QUESTIONNAIRE ^ 
AUTOMATED INST.RUCTIONAL MANAGEMENT SYSTEM (AIMS) 

( 



.PLEASE CHECK THE APPRGPRIAfE BLOCK TO 
INDICATE YOUR OPINIOM OF THE FOL^LOWING 
STATEMENTS. COMMENTS EXPLAINING VOUft 
SELECTION WILL BE APP'lECIATED. 

^ • • 


STRONGLY AGREE 


AGREE 


NEJTRAL 


Ul 

UJ 
CC 

< 


iiii 
m 
< 

O 

> 
-J 

o 
z: 
o 
ac 

(0 


COMWLNTS 


/ 


• 
• 

21. The instructors can 
answer my questions. 






* 




/ 




22^ There is a good working 
relationship betv/e^n the 
instructors and myself. 














23 4. Background noise (voices, 
movement, operation of 
equipment) is distracting. 


1 
1 


! 





1 


1 




24. Working in the carrels 
and ot)ier student posi- 
tions becomes tirefjome 
over a long period of 
time. ^ ' ^ 

• 














25. There are so many devisees 
(computer terminals, TVY * 
; cassettes / slide projjee^ 
tors) to opi^rate that it 
distracts from the in- 
'struotion.. 












4 

\ 

J 



' 82 



STUDENT ATTITUDE QUESTLONNAIRE 
AUTOMATED INSTRUCTIONAL MANAOEHENT SYSTEM (AIMS) 



PLEASE ChtCK THB APPROPRIATE 8L0CK TO 
INDICATE YOUR OPiNIOM OF THC FOILOWING 
STATEMENTS. COMMENTS £r,PlAININ(S YOUH 
SEISCTJON WILL BB 'APPRECJATEO. ) 

/ 


STRONGLY AGREE 


AGREE , 


NEUTRAL I 


DISAGREE 


STRONGLY DISAGREE 


COMMENTS 
















26. Constant changeij from 

one instructional medium 

to anothjer interferes 

vith learning. 
« 






■ 

f 








'21. The co0iput-«ir terminals 
are inoperative too 
often, which wastes 
my time. 








* 




• 


28. The computer terminal 
easy to operate. 












A' . 


29. The computer terminal 

text displays are clear 
.^nd e^sy to read. 














30. The amount of ^riat^r'ial 
presented on the indivi-^ 
du *1 terminal displays 
is not excessive. 

.ur^^ 




— 











STUDENT ATTITUDE? QUESTIONNAIRE 
AUTOMATED INSSRUCTIONAL MANAGEMENT SYSTEM (AIMS) 

ir" "~ ^ 



PLEASE CHECK THE APPROPRIATE BLOCK'TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 


STRONGLY AGREE 


AGREE 


NEUTRAL 


DISAGREE, 


STRONGLY DISAGREE 


COMMENTS 

1 '■ 


31. Graphic displays c n. tne 
computer terminal are 
sharp and easy to under- 
stand. 














32. I had no problem in 
learning to use 'the 
keyboard at the computer 
terminal . 










- 


• 


SECTION II. CMI Mode (Material 
^ presented -of f "line 
under computer 
managfement) . 

33. I prefer receiving all my 

assignii\ents via the com- 
' puter terminal. 














34 ♦ Most of my time in the 
course is spent in the 
CMI mode. 














35* The inst:;uctors kept 

\ referring me to computer 
terminal for directions 
rather than answering 
my questions about Off- 
line assignments* 















STUDENT ATTITUDE- QUESTIONNAIRE 
AUTOMATED INSTRUCTI^ONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED, 


1 STRONGLY AGREE 


1 AGREE 1 


\ NEUTRAL 


DISAGREE 1 


1 STRONGLY DISAGREE 


•V, 

. COMMENTS 

* 

/ 


36. I had no difficulty in 
• getting to use a compu- 
ter terminal whenever 
it was necessary. 


1 










• 


37. I feel that the AIMS 

instruction is too im- 
personal with so much 
time spent in the CMI 
mode. 


/ 












38. Computer down time did 
not affect my progress 
through the course. 














39 « It was difficult for me 
to determine my next 
training task when the 
computer system was down\ 




\ 






I 




40. You always know exactly- 
where you stand in the 
course when in the CMI 
mode . 















STUDENT ATTITUDE QUESTIONNAIRE 
AUTOMATEE), INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOpK TO 
. INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION W'ILL BE APPRECIATED. 


STRONGLY AGREE 


AGREE . 1 


NEUTRAL 1 


DISAGREE 


STRONGLY DISAGREE 


COMMENTS 


* 

41. like the feeling of ' 

independence associated 
^ with cornr'uter-mahaged 
'instruction. { 


• 












42. I would rather receive my 
off-line assignmen±s from 
an instructor so I can 
ask questions and clarify 
any points that I don't 
understand. 














43. I would like to have more 
instructior? in the CMI'* 
mode. 






r 








44. I would leather go through 
the course at the same 
' speed as the other stu- 
dents ds I seem to learn 
fhore in a group. 

— ^ jlS. 


V 










k 


45. ^It is easier to. connect 
learning elements in the 
CMI mode ,than it is under 
other types of instruc- ^ 
tion I have experienced. 

4f 












• 

" \ . 



STUDENT-^TT-ITUDE-eUESTIONNA-IR^^^^ 

AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM' (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION QF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 



46* I have experienced no 
problems-cj^ih moving 
through the course in 
CMI mode'. 



47. Instruction received via 
the .computer terminal 
are qleary concise/ and 
easy to follow. 



COMMENTS\ 



48, 



^I-feel th^t my progress 
would be faster if my 
course activities were 

'controri.ed by the class- 
room instructor. 



49. Not being able to ac»k 

the instructpj; a '^question 
immediately when a prob- 
lem occurred has made 
^ learning more difficult 
^ for me. \ ' 



50* 



Th^re Mete net enough 
computer terminals in . 
the course to meet all\ 
the student requirementis , 



T 



9i 



1 
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STUDENT -ATTi^0DE-Q»ESTi-6NNA-PRE 

• AO'l^OMATED .INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE Bl,OCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
statement's. COMMENTS EXPLAINING YOUR 
SELECfjON WILL BE APPRECIATED. 


1 STRONGLY AGREE 1 


Jagree I . I 


1 NEUTRAL 1 


1 DISAGREE 1 


1 STRONGLY DISAGREE { 


\ 
\ 

COMMENTS 


51 • Too much waiting occurred 
because the computer ter- 
minals were down for 
maintenance, ' ^ 




\ 


\ 








52. Waiting to get ^ computer 
terminal caused me to 
lose time in getting 
through a course, / 

—. V 


- 


\ 


; 








53, I would prefer to hav^ 
a computer terminal aft 
each classroom jpositioh. 

3 ■■ 






9 






* 

« 


54. In my course 'there, were 

too many students 'for the 
number of computer termin- 
*^ als available. 














55. The best configuration for 
the AIMS is to cluster the 
computer terminals. in one 
classroom. 

\ 


\\ 








- 1 


\ 

\ 



* 
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COMPUTER-MANAGED INSTRUCTION ATTITUDE QUESTIONNAIRE 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 




( gdMPUTER-MJ^AGED INSTRUCTION ATTITUDE. QUESTIONNAIRE 
AUTOMATED .INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) " 

The purpose of the following questionnaire is to col 
lect student opinions about Computer-Managed Instruction 
(CMI).' There are no right or wrong answers. Rather, we 
are ■ interested in your candid opinion.' Individual re- 
sponses will be held in strict confidence. 



) 



NOTE: - The Pretest Questionnaire is designed to be admin 
is.tered prior to the -Chi course and the Posttest 

Question;^aire is designed to be administered at 

the conclusion of the co'urs^. 
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PRETEST COMPUTER-MANAGED INSTRUCTION QUESTIONNAIRE 
CIRCLE THE LETTER OF YOUR CHOICE FOR EACH^ ITEM. 

/ 

1. When I am trying to learn something, it is important 
to mie to know where I stand in comparison to others. 

(a) Strongly ^ree 

(b) < Agree ^ J i 

(c) Undecided < ^ \ ' ' 

(d) Disagree" <^ j . . 

(e) Strongly disagree . / • i -v. 

^ >, ' 

2. I would like to take a CMI (Compjjter-Managed Ins true 
tion) course. * " . 

(a) Strongly agree • • • 

(b) .Agree ^ 

(c) Undecided > ' 

(d) Disagree / ^ ^ 

(e) ^trongly .disagree 

3. -Taking a CMI course would make me nervous. 

(a) Strongly agree 

(b) Agree • ; - 

(c) Undecided ^ 

(d) Disagree 

(e) Strongly disagree 

4. Taking a CMI course would be more interesting^ than 
staking the same cou^rse taught in some other way. 

(a) Strongly agree 

• I .(b) Agree 

. (c) Undecided 

^(d) Disagree 

\ (e) Strongly disagree 

5. : People ^shouJrl be taught by -other people, not 'by 

machines. ' ' v 

(a) , Strongly agree 

(b) Agree i 

(c) Undecided 

(d) Disagree 

(e) Strongly 'disagree 



V 
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6. I feel very uncomfortable when others know that I've 
made a mistake. 

' (a) Strongly agree 

, (b) Agree 

(c) Undecided \^ 

(d) Disagree ♦ 
(e> Strongly disagree 

7. I think I would feel isolated and a^one while taking a 
. CMI course. ^ . 

. (a) Strongly agree " 

(b) Agree 

(c) Undecided - 
♦ » (d) Disagree^ 

(e) Strongly disagree 

r » 

8. I like lit when t can immediately find out where I have 
. made my mistakes. 

(a) Strongly agree 

(b) Agree 

(c) Undecided 

(d) Disagree 

(e) -Strongly disagree 

9. It would be boring to take a CMI course. 

(a) Strongly agree 

(b) Agree 

(c) Undecided 

(d) ' Disagree 

(e) Strongily disagree 

10. I think it would be easy to understand the material in 
; a Cmi course. . - ^ ^ 



(a) Strongly . 3e 

(b) Agi.ee 

(c) Undecided 

(d) Disagree 

(e) Strongly disagree 



11. Students are being treated more and more like IBM cdrds. 



(a) Starongly .agree 

(b) Agree 

(c) Undecided 

(d) Disagree 

(e) Strongly disagree 
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12. Taking a CMI course would be too mechanical. 

(a) Strongly agree 
^ (b) . Agree 

(c) Undecided 

(d) ^ Disagree 

(e) Strongly disagree 

•^13. I think I would feel challenged to do my best work 
while taking a CMI course. 



(a) All the time 

(b) Most of the time 

(c) Seme of the time 

(d) Only .occasionally ' * 

(e) Never 

14. I don't like to have my errors pointea out to me. 

(a) Strongly agree 

(b) Agree 

(c) Undecided 

(d) Disagree 

(e) Strongly disagree 

15* I would prefer^ to have most courses taught as CMI 
rather than by other teaching methods. 

(a) Strongly agree 

(b) Agree 

(c) Undedided 

(d) Disagree 

•(e) Strongly disagjree 

16. Most courses could be managed more effectively by a 

regular teacher than by computer. 

>^ 

(a) Strongly agree- 

(b) Agree 

(c) Undecided 

(d) Disagree 

(e) Strongly disagree • 

/ 
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POSTTEST COMPUTER-MMIAGED INSTRUCTION QUESTIONNpRE 

I 

CIRCLE THE LETTER OP YOUR CHOICE FOR EACH ITEM. ' 

1. The way the material was presented to me made me feel 
that- no one really cared whether I learned or not. 

(a) Strongly agree 

(b) Agree 

(c) Undecided 

• \ ' (d) Disagree ^ 

(e) Strongly disagree ^ 

2. I was npt concerned when I missed a question because no 
. , one was watching me - anyway. 

(a) Strongly agree 

(b) Agree 

(c) Undecided t 

(d) Disagree 

(e) Strongly disagree 

3. • The method by which I was told whether I had given j a 

' right or wrong answer became boring. 

(a) Strongly agree 

(b) Agree 

(c) - Undecided 
/ (d) Disagree 

(e) Strongly disagree 

4. I was concerned that I might not be understanding the 
material. 

r 

(a) ' All the time 

(b) Most of the time 

(c) Some of the time 

(d) ,Only occasionally 

(e) ^JNever 

5. The responses to my answers were appropriate. 

^ (a) All the time 

\ (b) Most of the time 

' . (c) Some of the time 

(d) Only occasionally 

. (e) Never ^ ' " 
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6. I felt, uncertain as to my performance compared to the 
performance of others. 



(a) All the time ^ 

(b) Most of the time 

(c) Some of the time 

(d) Only occasionally 

(e) Never" 

7. I knew whether my answers were correct or not before 
I was told. 

(a) Quite often 

(b) Often 

(c) Occasionally 

(d) Seldom 

(e) Never 

» 

8. I found myself jus£ trying to get through the lesson' 
rather than trying to learn. 

^, * (a) All the time * 

(b) Most of the time 

(c) Some of the time 

(d) Only occasi'^nally ^ 

(e) Never 

9. I guessed at the answers to auestions* 

(a) Qviite often 

(b) Often 

( c ) Occas ionally 

(d) Seldom 

(e) Very seldom 

10* I was able to work at my own pace. 

(a) All the time 

(b) Most of the time ^ 

(c) Some of the time 

(d) Only occasionally 

(e) Never 

11. I was aware of efforts to suit the material specifically 
to me. 

(a> Qv^ite often 

(b) Oft^n 

(c) Occas^ionally 
(<a) Seldom . 

(e) Very seldom . . . ^ 
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12;. I found it difficult ta concentrate on the course 
material because of the machx.e. 

, (a) All the time 
(b)^ Most of the €ime 
•(c) ' Some of the time 

(d) , 0nly- occasionally 

(e) Never , 

13; Computer-managed instruction made it possible for me 
' to learn quickly. 

{a) Strongly agree 

(b) Agree 

(c) Under^ided ' 
, (d) Disagree 

(e) Strongly disagree 

14. ' Questions were asked which were not .rele^^'"^ t to the 
material presented . 

• ^ (a) AIL the time 

(b) 'Most of the time 

(c) Some of the time 

(d) Only occasionally 
.(e) ^Never 

» • 

15 • Based upon my expogrience v^ith this course, I prefer 
CMI to other methods of instruction* 

(a) Strongly agree 

(b) Agree 

(c) Undecided 

(d) Disagree 

(e) Strongly disagree 

16. When I am trying to learn sometfiing, it is important 
to^me to know where I stand in com{>T*rison to others* 

(a) Strongly agree - 

(b) Agree 

{c) Undecidea ^ ^ 

(d) ►Disagree . 

{^) Stijongly disagree 

17* T would like to take another CMI course, 

\ (a) • Strongly agree^ 
(b) Agree 

^ (c) ' Undecided ^ , * 

Cd) Disagree \ ^ 

(ej Strongly disagree 



18. Taking a CMI course made me nervous. 

(a) Strongly agree 

Cb) "Agree v.* 

* (c)* Undecided 

• (d) Disagree • . 

(e) Strongly disagx:ee 

19. The material managed by computer was more interesting 
than taking similar material managed in some other 
way. 

(a) Strongly agree 

(b) Agree 

(c) Undecided ' ; 

(d) Disagree 

(e) Strongly disagree 

20. People should, be taught by other people, not by 
machines . 

(a) Strongly agree 

(b) Agree 

(c) Undecided ^ 

(d) Disagree 

- (e) Strongly disagree 

21. I feel Very uncomfortable when others know that I've 
made a mistake. 

(a) Strongly agree 

^^y J^gy^^Q^ 

(c) Undecided 

(d) Disagree 

(e) Strongly dis*agree 

• ^ 

22. I felt isolated and alone" while working with the 
: computer. 

(a) Strongly agree 

(b) Agree , 

(c) Undecided 

(d) Disagree • 
\e) Strongly disagree 

23. i liked it when. I was able to find out where I had 
made my mistakes. 

(a) Strongly agree 4^ 

(b) Agree 

(c) Undecided . ' 

(d) Disagree ' . • . 

(e) * Strongly disagree 
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24. It was baring to learn material managed by/ computer. 

^ (a) , Strbngly agree^ 

fb)' Agrfee , ; 

(c) . Undecided 

(d) Disag"ree 

(e) Strongly ^isagree^ 

'25, It was easy to understand the CMI cours^ material. 

(a) Strongly agree 

(b) Agree 

(c) Undivided 

(d) Disagree 

(e) Strongly disagree 

26. Students are being treated more ;and mor^ like IBM cards. ^ 

^ {a)'7 Strongly agree f 
V (b) Agree 

(c) Undecidedi 

(d) Disagree 

(e) Strongly disagree 

27^. Courses managed by a computer H^ere too mechanical. 

(a) Strongly agree 

(b) Agree 

(c) Undecidfd 

(d) Disagree 

(e) Strongly disagree 

28. I felt challeng^ed to do my best work while in a CMI 
course . 

(a) All the time a 

(b) Most of thfe time 

(c) Some of the time 

(d) Only occasionally 

(e) Never 

29. I don't like to have my errors pointed out to me. 

\bl) Strongly agree ' > ^ , • 

(b) Agree 

(c) Undecided • 
{d) Disagree • 
(e)/ Strongly disagree 
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30* I would prefer to have most courses /taught as CMI 
' rather than, by other teaching laethoids ♦ ^ 

^ (a) Strongly agree 

(b) Agree 

(c) Undecided 
(d} Disagree 
(e) Strongly disagree 

Most courses could be managed more effectively by a 
regular teacher than by computer. 



31. 



(a) 
(b) 
(c) 
(d) 



Strongly agree 
Agree 
Undecided 
Disagree 
Strongly disa^gree 
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Attachment 9 



' • COST CATEGORIES ^ 

Automated instructional management SYSTE^^ (aims) 



This list indicates th,e categories of costs incurred 
for Coinputer-Based Training Systems. It is propose^ that 
the Seidel and Wagner (1977) Cost-Effectiveness Specifica- 
tion be used for the collection of this data on the AIMS. 
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COST CATEGORIES 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



Categories 




1. Equipment 


i.i 


Compu^ter (s) 


1.2 


Terminal (s) • 


1*3 > 


Auxiliary AV Devices 


1*4 


Auxiliary Memory 


1.5 


Locdl Interfaces 


1-6 


Telephone Lines 


1.7 


Special Lines 


^ 1.8 . 


Satellites 


1.9 


Receiv^r^? 


l.iO 


Power Generating Equipment 


1.11 


Carrels 


1.12 ' 


Other Equipment 



2. Facilities " 

•2.1 Classrooms 

2.2 • Laboratories , < 

2.3 Large Group Instructional Spaces 
2.4* Offices 

2.5 • Individual Learning Spaces 

2.6 Libraries and Other Information Resource 

_^_....__-..,-,..^enters-*- 

2.7 Other Facilities * . 



3. Software 

3.1 Systems Programs 

3.2 General Applications- Programs 

3.3 Diagnostic/Test Programs 

3.4 Utility Programs 

3.5 Other Computer Progr^s 



4. Instructional Systems Development* 

4.1 ^ Analyze (Phase I) 
;4.2 Design Phas^ (Phase if) 
.4.3 Development Phase (Phase III) 
4.4 Other Inst^uqtional, Systems Development 
(ISD) • Activities 



*If CMI Subsystem is Operational 
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COST CATEGORIES 

AUTOMATEp/tBlSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 

© * 

5. Instructional Methods/Materials* 

5.1 Audio ^ 

5 . 2 Audio/Visual • 

5.3 Film Text/Visual 

S'.4 Lectiure/Dembnstfatioxx 

5.5 Group Discussion/Seminar 

5 . 6 Performance/Practice 

5.7 Tutoring {Peer or other) 

5.8 Printed Test/Visual' ' 

5.9 Other Instructional Methods/Materials 

5.10 Tests 

*If CMI Subsystem is Operational 

6. System Management/Test 

» _ ■-- ' , 

6.1 System Integration Engineering 

6.2 Program Management 

6.3 ' Operational Test 

, 6.4 Other Direct Management Costs 

7. Other Direct Cost^s (not included in the above) 

7.1 Supplies 

7.2 Travel 

7 . 3 Consultants 

7 . 4 Contracts/Subcontracts 

7.5 Other Direct Costs 



8. Preparation 

8.1 Training of Initial Site Personnel Cadre 

8.2 Site Personnel Pay and Allowances 

8.3 Other Preparation Costs 

9. Acceptance Test/Management 

9.1 Program/Project Management 

9.2 Acceptance Test 

9.3 Engineering Changes 

9.4 Site Checkout/Activation 
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Equipment 

10 •! Replacement ^Spares andj. Repair Test 

Equipment ' ^ ^ 
10 . 2 Other- Equipment 



System Management 

11*1 ' - Program/Project Management 
11.2 Other Direct Management Costs 



Other Effectiveness Measures 

12.1 .Time Measures 

Average Training rime in Course.^ 
. Average Testing Time in Course 
Average Course Time - 

12.2 Achievement Measures (Final 'Criterion 
' Test Results-First Attempt) 

Accuracy or Speed Scores 
Gain Scores 

Niimber of Objective^ Passed 
% Students Passed 
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INSTRUCTOR TASK LIST . 
AUTOMATED INSTRUCTIONAL JMANAGEMENT "SYSTEM (AIMS) 
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^ ^ INSTRUCTOR TASK LIST 

irOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



This task list describes the activities that an in- 
structor may be performing in courses managed by the AIMS 
It can serve as the. basis for a -checklist to be employed 
when determining' the impact of AIMS' upon the job of in- 
structor, at USAFAS. 
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INSTRUCTOR TASK LIST . 
AUTOMATBIV INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



ASSIST STUDENT: Greet student, call roll, inform student 
of classroom procedure, update studfent records, answer 
si:udent's questions ♦ 

CONDUCT COURSE SUPPORT: Check serviceability of lesson^ 
ma'terials, and equipment. Store,, organize and inventory 
lesson materials, tools, and equipment. Secure classroom, 
review and update lesson material. Maintain technical 
orders, maintain classroom expendable supplies* ' 

OVERSEE MEASUREMENT CENTER: Provide examination, material, 
update student record, grade/critique test, assign student, 
and monitor test center. 

CONDUCT STUDENT SUPPORT: Maintain student records, sched- 
ule/monitor .individual assistance and student breaks. Give 
safety briefing, enforce safety, insure proper classroom 
management, lightimg and , temperature', conduct preassessment 
briefing and orientation* Monitor student progress* 

CONDUCT OTHER SUPPORT: Maintain unclassified file and pub- 
lication file. , Review incoming distribution, inspect 
faciLtty, receive and assist visiting official, attend 
meetings, 

CONDUCT GRADUATION^ Conduct final critique, figure time 
in course and ^ grade average, and conduct ceremony* ^ • 

CONDUCT PEMONSTRATION/PERFORMANCE: Conduct demonstration/ 
performance to give student hands--on training on equipment 
related to written material. • 

COUNSEL/CRITIQUE STUb'SNT: Counsel ^tudent on progression 
an^ imjD.rovement areas. Couhsel and assist students with 
morale, welfare and disciplinary problems. Take necessary 
corrective action required to maintain -discipline. - 

REVIEW COURSE MATERIAL: Review new and old course and 
test material. , ^ ' ' 

GftADE TESTS: Review and grade tests, inform student of 
grade and area needing review*^ 

UPDATE LESSON MATERIAL: Update lesson material to reflect 
changes in controlling document 

DEVELOP NEW COURSE MATERIAL: Develop new course and test C 
material* 

* > 
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INSTRUCTOR TASK LIST 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 

OPERATE AIMS TERMINALS: Request records and reports , ob- 
tain computer prescription, override computer, request stu 
'dent ^ata<^ locate Btudent, update student record, update 
resource availability # 

REQUEST MAINTENANCE ASSISTANCE: . Contact major source for 
repair. Maintains log of when assistance was called for 
and when corrective action was tajcen. Maintains log of 
down time. 

CONSTANT MANNING REQUIREMENT: Monitor classroom. 
OTHERS 



Prom: Dallman/ et al*, 1979. 
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INSTRUCTOR ATTITUDE SURVEY 
AUT<mTED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



\ 



J 

\ 
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INSTRUCTOR ATTITODE SDRVElf 
AUTOMATED D?STROCTIOl?AI» MANAGEMENT SYSTEM (AIMS) 



The. purpose .of this survey is to determine your 
activities and opinrdns regardingnitie various aspects of 
the AIMS and Computer-^^naged Instruction -(CMI) . Vlease. 
answer ail items in th^ survey with complete candor. Your 
individual responses will be^held in strictest confidence/ 



PART X: Background 

1. - Course ; 

2. * Date; \ 

3. How long have you' been an instructor in this coarse 
(months)? 1_ 

4* Previous instructor position (s) was in a: {circle 
appropriate choice (s)) 

a. Conventional classroom environment j 

b* 3elf**paeed tnuitimedia instructional enviromtjent • ^ 

Computer managed self -paced instructional environs- 
men t» 

No previous instil tor experience^ 
e. Others {speci£y) " ^ , 





4' 



PART 11^ A^'ti.bud6S 
AUTOMATED INSTRUCTIOm MANAGEMENT SYSTEM (AIMS) 



CHECK APPROPRf ATE BLOCK TO INDICATE YOUR 
ATTrrUDE TOWARDS THE AIMS.. EXPLAIN YOUR 
SELECTION IN THE COMf^ENTS SECTION AS 
NECE^SSARY. 

1 

• * 


STRONGLY AGREE 


ill 

Ul 

cc 
o 
<x 


1 NEUTRAL 


j DISAGREE 


STRONGLY DISAGREE 


COMMENTS 


1* The majority of the students 
are meetijig the objectives . 
of the caUrse .using ftiis 

' method of instruction. 

r 










« 




2.' yhis method is not an effec- 
, , tive way_ to instruct this 
^ course-. 


/ 




/ 






r 


-* * 

3. The examinations adequately 
evaluate th6 students on 
the achievement of training 
'objectives. 












• 


■ >; ■ ' 

4* The examination procedure 
is inconvenient and- takes 
too much time, 

\* 

< 




• 








* 


$• This n^ethod of instructipn 
is a valuable teaching pro- 
^ "cedure*; • 


V 

#* 
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PART II: • Attitudes (Cont) 
' AUTOMATED -INSTRUCJIONM. MANAGEMENT SXSTEM (AIMS) 



* - ..... — ■ ' ' 1 

CHECK APPROPRIATE BLOCK TP JNOICATE YOUR 
ATTITUDE TOWARDS THE AIMS. EXPLAIN YOUR 
SELECTION IN THE'COMMENTS SECTION AS 
NECESSARY. ' 


Ui 

cc 
o 
< 

o 
o 


AGREE . \ \ 


-J 
< 

tc 
1- 
z> 

Ui 


DISAGREE ^ 


STRONGLY DISAGREE \ 


COMMENTS ' 


6. It takes *an experienced in- 
structor to 'manage this type 
of instruction.. 


• 










• 


7. The students' attitudes to- 
ward the course are very 
favorable* 










- 


• 


8. The students use too much 

time learning to operate in-* 
structional equipment (other 
than computer terminals) • 














9, Students appear to be very 
interested in the course 
content • 


1 












10 • The students are very en- 
thusiastic about the 
course • 

# 















•X— ^ 
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PART At-titudes (Cont) 

. AUTOMATED INSTRUCTIONAL. MANAGEMENT SYSTEM (AIMS) 



CHECK APPROPRIATE BLOCK TO INDICATE YOUR 
ATTITUDE TOWARDS THE AIMS, EXPLAIN YCUR 
SELECTION IN THE COMMENTS SECTION AS . 
NECESSARY. ^ 


• STRONGLY AGREE 


AGREE 


NEUTRAL 


{£, 

< 

CO 

o 


STRONGLY DISAGREE 


COMMENTS 


il* The students tend to study 
less as the instruction 
progresses* 














12. I have no difficulty answer- 
ing students' questions. 






• 






* 


13 * I have developed a good 
















working relationship wj^tlr- 

the students. 












14. I use my instructor time 
more efficiently in this 
w method of instruction 














15. Tn is course has numerous 
learning elements that do 
not, lend themselves to 
computer-managed instruc- 
tion. 

v 















PART .lit Attitudes^ (Cont) 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



^ 

CHECK APPROPRIATE BLOCK TO INDICATE YOUR 
ATTITUDE TOWARDS THE AIMS. EXPLAIN YOUR 
SELECTION.IN THE COMMENTS SECTION AS 
NECESSARY. - 

-/ 


STRONGLY AGREE 


AGREE 


NEUTRAL 1 


DISAGREE 


1 STRONGLY DISAGREE 


COMMENTS 


16. The AIMS teaches as well 
as other methods, and in 
less time. 










• 


• 

* 


17 • The instructional proce- 
dures in this course need 
a major revision. 






• 


; 






18. Instructional materials 
must be more thoroughly 
organized then for group- 
paced teaching methods. 


u 










« 


19. The instructional materials 
concentrate on specific 
skilTs^and- knowledges, and 
excludes "^nice to know" 
lesson material. 












o 
















20. Th'e instructional materials 
and fmedia are suitable for 
the type of student input 
into this course. 












> 

1 



' 123 

113 



*PART II: Attitudes (Coht) 
AUTOMATED INSTOUCTIONAL MANAGEMENT SYSTEM 



(AIMS) 



CHECK APPROPRIATE BLOCK TO INDICATE YOUR 
•ATTITUDE TOWARDS THE AIMS. EXPLAIN YOUR 
SELECTION IN THE COMMENTS SECTION AS 
NECESSARY. 



ur 
u> 
oc 
O 
< 



1 



COMMENTS 



21. Using this method the in- 
structor to student ratio 
can be reduced ovfer ^roup- 
paced methods. 



22. 



Students experienced litt;J,e 
difficulty in operating tlhe 
computer terminal. 



23. Computer-managed instruc- 
tion will enable a rapid 
increase in student inputs 
should need arise. 



24 ♦ Looking to the. future^ I 

£eel that CMI will replace 
other methods of instruc- 
tion. ♦ 




PART II: Attitudes (Cont) 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM CAIMS) 



CHECK APPROPRIATE BLOCK TO INDICATE YOUR 
ATTITUDE TOWARDS THE AIMSi EXPLAIN YOUR 
SELECTION IN THE COMMENTS SECTION AS 
NECESSARY/ ' . 



26. Any capable instructor can 
prepare lesson materials for 
CMI courses. 



1X1 

cc 
o 
< 

Q 
>- 

z 

o 

cc 



COMMENTS 



27. All instructors should .take 
a CMI course prior to teach- 
ing in this. mode. 



V ' lis 
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PART II: Attitudes (Cont)' 
AirrOi^ATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



23 • Do you feel that the AIMS has done a 
good job of managing the students off- 
line activities. Please explain* 



29; Has there been an oppositlDn--i:crTVI^ 

among the instructox»?^''''lf yes^ please 
explain. 



- 30. Do you think that the CMI aspect of 

the AIMS has accelerated the students' 
ji; ' progress through'^the course?^ 
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PART III: Activities (Cont) 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



HOW MUCH OF YOUR TIME DURING 
INSTRUCTIONAL DUTIES FOR THIS OOUnbt 
DID YX)U SPEND ON EACH OF THE 
FpLLOWING ACTIVITIES? CHECK THE 
MOST APPROPRIATE RESPONSE AND 
EXPLAIN RESPONSE IN COMMENTS 
AS NECESSARY. 


MORE THAN HALF 


SUBSTANTIAL AMOUNT 


1 FAIR AMOUNT < ^ 


VERY LITTLE ' 


NONE 


COMMENTS 


31. Monitoring, students. 
• 














32. Individually counseling 
students on lesson 
material. ' . 




- 








• 


33. Counseling students in 
small groups on lesson 
material. 














34*. Administering tests. 


V 










• 


35. Giving direction on 

performance exercises. 














36. Maintaining student 
records . 


\ 
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PART III: Activitie^s. (Cont) • . ' 
' AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



HOW MUCH OF YOUB TJME DURING . 
INSTRUCTION)\L DUTIES FOR.THIS COURSE 
DID YOU SPEND ON EACH OF THE 
FOLLOWING ACTIVITIES? CHECK THE 
MOST APPROPRIATE RESPONSE AND 
• EXPLAIN RESPONSE INh COMMENTS 
AS NECESSARY. 


MORE THAN HALF 


SUBSTANTIAL AMOUNT 


L 

Z 

ZD 

o 
< 

cc 
< 


VERY LITTLE ' 


NONE 


COMMENTS 


37. Putting malfunctions in 
equipment for perform- 
ance exercises. 




-. 










38. Checking questions^ per- 
formance problems, or 
exams. 






• 








39. Preventive maintenance 
on equipment ; 










• - 




40. Repairing faulty equip- 
ment. 

« 












9 


41. Reviewing students* 
* records. 












* 


42. Settin-g^^ and giving- 
guidance on nset of in- 
^ structional media. 















lis ^ 



PART III: Activities (Cont) 
AUTOMATED INSTRUCHONAL MANAGEMENT SYSTEM (AIMS) 



how much of your time during 
instructional duties for this course 
did you spend on each of the 
Following activities? check the 
most appropriate response and 
explain. response in comments 
as necessary. 



43. Counseling students 
on matters other than 
lesson laaterial. (Career 
guidance) 



COMMENTS 



44. Recording instructor 

comments about students. 



45* Faculty bdard actions. 



46. interaction with other 
instructorn. 



47. Administrative duties 
not mentioned. ,(List 
under comments if they 
take luuch of your .time.) 



48. Preparing for classroom 
instruction. 



ERLC 
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PART III: Activities (Cont) 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



• 

HOW MUCH OF YOUR TIME DURING 
INSTflUCTinNAI DUTIP^ POR THl'^t PniJR^P 

DID YOU SPEND ON EACH OF THE 
• FOLLOWING ACTIVmeS? CHECK THE 
MOST APPROPRIATE RESPONSE AND 
EXPLAIN RESPONSE IN COMMENTS 
AS NECESSARY. 


MORE THAjN HALF 


SUBSTANTIAL AMOUNT 


FAIR AMOUNT 


VERY LITTLE 1 


NONE 


COMMENTS 


4 

49, Self improvement • 












• 


t 

50. Others (List under 
comments) • 

v. 














• 






• 






• 














• 


* 

* 














f 
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. PART Illt Activities (Cdftt) 
' AUTOMATED rNSTRUCTIONAL MAMGEMENt SYSTEM ' (AIMS) 

51, WhicU of the activities do you feel take too much of 
your instructor time? Indicate by number from the pre- 
ceding list (Items 31-50) • 



.52. Which of. the activities do ^ou feel you need to de- 
vote more' of your instructor time to? Indicate by number 
from the preceding list (Items 31-50) . 



» \,^yPAKq^ IV: Suggestions/Observations 

53. the classroom environments you are familiar with, 
which of tl?e following would you prefer to be an instructor 
in? (Circle best response) 

a* Conventional classroom environment • 

b. Self -paced classroom environment. 

c. computer managed self -paced classroom environ- 
ment. 

d. Others (specify) 



54. "What training (if any) do you think is needed for 
, instructoirs using the XlMS instructional method? 



PAKT IV: Suggestions/Observations (Cont) 
-s AtrrOHATBD INSTRDCnOHAL MASAGEMENT SYSTEM (AIMS) 

55. What guidelines or. advice would you give a new in- 
structor who is going to use this teaching method fo.r 
the first time? 



^6,- Which of%he following items would you identify as 
Ivantages of CMX? (Rank in order of importance*) 



2^dvaj 



ji. Reduced training time. 
_b.^ Higher student achievement. 



_c. Better stud^rit motivation. 

_d. Lower failure rate. 

_e. Preferred by most students, 

^f. Fewer instructors required. 

_g. Greater staff and faculty interest, 

h. Better performance after training. 



i. Prc^tes development of the lemming process. 



j). An effective teacher-aide to the classroom 
instructor. 

k. Other (specify) 



57.. What are the disadvantages of this method of in- 
struction? 
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n^AR? IV: SuggestlcA/Observations ((kmt) 
AOTOHAIED IMSTSfircnOHAlJ MAKAG&EMT SSSIEH ( ^MS) 

58 • ij^at problems. have you encountered using this method 
of ir^struction? { 

59* If you v?>re the Director* of. Training? what would you 
do to itcprove^ the AIMS? 

r\ . / 

^60. What is your frank opijuion about 't^e relative worth 
^of computer--flianaged instruction? 



61* How has the implementation of AIMS assi^ed you in 
monitoring student progress in the course? Please explain. 



62. Do you feel^ that'using^ the computer to monitor stu- 
dent progress has, in fact/ been more effective than the 
method used previously? Please explain. 



63. Can you define x>r describe any unique or unusual ^rob- 
ifijas encountered in preparing instructional materials not 
"xj identified when preparing self-spaced material? 



64. What is your frank opinion of using the computer to 
randomly generate test items? > 



6S. Has the identification, recording, and hard copy 
summation of test performance been helpful in pinpointing 
weaJc areas of instruction? 
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PART IV: Suggestions/Observations (Cont7 
^ AUTOMATED INSTRUCTIOMAL MANAfeEMBST SYSTEM (AlMslr 

* ^ » • 

"^6* .What .protJleins^ if any, have you experienced, with 

student qveueing (either waiting to he tested or to 

start* the 'next task)? 



67.' Has the AIMS enabled the student to achieve his/her 
objectives in Ies6 time than the prior self -paced course 
format?.^ Yes] No ♦ 



*68. Do you find that, the AIMS student graduation predic 
tions have been more timely and functional than, those re 
ceived under the previous self-paced operation? Please 
explain. 





Attachment 12 




STAFF AND FACULTY ATTITUDE SURVEY 
• AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 






STAFF AND FACULTY ^MTITODE SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEidENT SYSTEM (AIMS) 



DATE 



This is a survey to gather information relative to' 
the Automated Instructional Management System (AIMS) • 
Please answer all items ^n tljis survey with complete 
candor. Your responses will be held in strictest con- 
fidence.. If you feel that you are not in a position to ^ 
answer a particular item because you have not been closely 
associated with the AIMS project, circle the item number 
and leave it blank. Your comments or suggestions .will be 
greatly appreciated. 

, Please place a check (ix^) mark by your position below. 

Instructor 



Education Specialist 
Training Specialist 



Section Chief 



Course Chief 



Division Chief 
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~ : STAFF .ANEL.E&C_ULT,Y SURVEY 

AUTOMATED "INSTRUCTIONAL MANAGEMENT^SYSTEM (AIMS)" 



PLEASe CHECK THE APPROPRIATC BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMWENTS EXPLAIf^ING YOUR 
SELECTION WILL BE APPRECIATED. 


1 STRONGLY AGREE . 1 


AGREE 


1 NEUTRAL . 1 


DISAGREE" 1 


STRONGLY DISAGREE 


* 

COMMENTS 


SECTION !• Attitude Toward ^ 
the AIMS 

!• The AIMS is an effiective 
instructional method 
which can produce high 
levels of achieAjgnent* 












• 


2. The AIMS m^kes, explicit 
use of the learning pro- 
cess. 




• 






> 




3. Generally^ student, reac- 
tion to the AIMS has been 
favorable. 



























4. The AIMS instructional 
materials must be better 
organized than other self- 
paced instruction. 












* 


5. The AIMS should produce 

a graduate better equipped 
to perform at job entry 
level than does other 
self -paced instruction. 












* 



■ ' STAFF AND- FACULTY SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOCK TO 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED, 


[strongly agree' 


AGREE 


i NEUTRAL 


DISAGREE 


STRONGLY DISAGREE 


• 

t 

COMMENTS 


6. The AlI^S teaches as well 
as other methods, and in. 
. less time. 

* 












- 


7. The ^IMS ccncentrates on 

specific skills and know-^ 
ledges and excludes "nice^ 
to know" items. 


• 










• 


8. AIMS' students are more 
apt to retain skills "aind 
knowledges learned for a . 
longer period ^af ter com- 
pletion. 














9. Under the AIMS program, 
students have varied 
repetition or- practice. 

, r 














• 10 . ' The AIMS instructional 
strategies are suitable 
for all types of stu- 
, , . dents found in the" 
courses. , , ^ 

• 




> 
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\ ' STAFF AND FACULTY" SURVEY 

AUTOMATED INSf RUCTIONAL MMIA6EMENT SYSTEM^ JAIMS) 



PLPASE CHECK THE APPROPRIATE BLOCK TO - 
INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENtS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 

♦ 


1 STRONGLY AdREE 


1 AGREE . / 


! NEUTRAL ' • I 


Ul 
Ul 

oc 
a 
< 


STRONGLY DISAGREE 


COMMENTS 


11. The AIMS instructional 
/materials are designed 
to fit all levels of 
students. . 




V 








• 


s 

. . 12. The AIMS courses do an 

.outstanding job o^ teach- 
ing students to meet 
performance objectives. 














13. The AIMS enables better 
. students to proceed at 
an accelerated rate and 
they invariably qomplete 
the course in less than 
programmed time. 














14. Under the AIMS, course 
lengths can be adjusted 
with minimum delay based 
on student progress. 












• 


15. .The AIMS .enables the' 
instructor to organize 
course materials more 
effectively to meet in- 
dividU'^il students needs; 
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STAFF MfD FACULTY SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT. SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOCK TO 


Ul 
Ul 

cc 
o 
< 








[DISAGREE 




INDICATE YOTJR OTINION OFTHE^FOLL-OWINQ- 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 


STRONGLY 


AGREE 


NEUTRAL 


DISAGREE 


STRONGLY! 


COMMENTS 


16. Slow learners, who would 
be academic failures in 
group-paced courses , are 
able to attain the mini- 
mx2|n performance standards 
under 'the AIMS. 












• * 


17. The AIMS provides more 

opportunity for students . 
^ to react to simulated 
on-the-job performance 
standards under the AIMS. 








• 


• 


1 


18. The AIMS provides a 

learning environment that 
matches the students 
Ijearnirig performance with 
materials appropriate to 
his learning rate. 
* » — < 










\ 


> * 


19. The greatest' value of 
the AIMS is that it en- 
ables the student to 
learn at his/lier own 
pace. 
* 












9 


20. The most valid criterion 
for judging the success 
. of "the AIMS is the stu- 
dents' performance after 
training. 

* 








1 
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'STAFF AND FACULTY SURVEY 
AUTOMATED INSTRUCT-IONAIc MANAGEMENT SYSTEM (AIMS)' 



PLEASE CHECK THE APPROPRIATE BLOCK TO 
-INDICATE YOUR OPINION OF THE FOLLOWING 
STATEMENTS. COMMENTS EXPLAINING YOUR 
SELECTION WILL BE APPRECIATED. 


1 STRONGLY AGREED 


AGREE ' 


< 

UJ 
Z 


DISAGREE 


STRONGLY DISAGREE 


COMMENTS 


21. Over a period of time/ 
. the novelty ef f ect--of . 
' the AIMS wears off and 
the ^udents tend to ^ 
become ^ bored . 














22. It is not anticipated 

tfiat the implementation 
of tho AIMS will have 
any impact on the man- 
' power structure of the 
courses involved. 














23. The administrative re- 
• quirements (maintaining ' 
studei)t records and re- 
ports) for the AIMS 
courses do not differ 
s greatly from other self- 
paced courses. 








: 






24. On-line pretests/post- 
tests bave-proved to be 
hiahly effective as a 
teaching/learning stra- 
tegy. 














25. The AIMS has significantly 
reduced the instructor to 
student ratio resulting 
in a saving of instructor 
personnel-, 












> 
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STAFF AND FACULTY SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS) 



PLEASE CHECK THE APPROPRIATE BLOCK TO 
" |NDICATE-Y0UR-0PiNIQN-0F-TH6^F0LL0WING 
, • STATEMENTS. COMMENTS EXPLAINING YOUft 
SELECTION WILL BE APPRECIATED. 

* . ■ 


STRONGLY AGREE 


AGREE 


NEUTRAL 


DISAGREE 


STRONGLY DISAGREE 


\ 

COMMENTS 


26. To be effective r instruc- 
tors must be trained in • 
the AIMS techniques in 
order to properly prepare 
themselves to meet stu- 
dent needs* 




























27.' Except for adding another , 
instructor medium, the 
implementation of the 
AIMS has caused little 
or no change in the 
course operation. i 






- - 




-~ 


\ 

\ 
\ 


u 

» 














• 

i 












V 


• 

• 

♦ 
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STAFF AND FACULTY SURVEY 
AUTOMATED INSTRUCTIONAL MANAGEMENT SYSTEM (AIMS)' 



28* The time required for development of AIMS instructional 
l^aterials - . 

a» exceeds that required for group-paced instruction . 
by (indicated percentage) %• 

b* is about the same as that required for grOup- 
paced instructibn . 

' c* is less than that required for group^paced in- 
struction by (indicate percentage) %• 



SECTION II* Open-Ended Questions • 

29. ^ In your opinion, what are the advantages and disad- 
vantages of the AIMS instruction when compared to *b^ther 
(group-paced/self -paced) instructional _ro^thpds?> • 



Advantages 



Disadvantages 



30. If you were in a position to dictate policy for future 
self -paced courses in other service schools, what initial 
guidelines would^you give your project officer? 



r 



31. What is your frai^k opinion concerning the relative 
worth of self-paced versus group-paced instruction? 
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